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Jed Margolin 3570 Pleasant Echo Dr. 

Phone: (408) 238-4564 Email: jm@jmargolin.com 



San Jose, CA 95148-1916 
June 17, 2003 



Mr. Alan J. Kennedy 

Director, Infringement Division 

Office of the Associate Genera! Counsel 

National Aeronautics and Space Administration 

Headquarters 

Washington, DC 20546-0001 
Attn: GP(02-37016) 



Dear Mr. Kennedy, 

I have received your letter dated June 1 1 , 2003. 

In my contacts with NASA personnel I have repeatedly stressed my desire that this matter be 
resolved in a friendly manner. However, since NASA has rejected my request to consider a license 
proffer and in view of your letter of June 11, it is clear that NASA has decided to handle this in an 
adversarial manner. 

Before I respond to your letter in detail, I want to make things easier for me by withdrawing my 
U.S. Patent 5,566,073 Pilot Aid Using a Synthetic Environment from this administrative claim in order 
to focus more directly on NASA's infringement of my U.S. Patent 5,904,724 Method and Apparatus 
For Remotely Piloting an Aircraft . However, I reserve the right to file a claim concerning the '073 
patent at a later time. 



(J) The identification of all claims of the patent(s) alleged to be infringed 

As I stated in my email of May 13, 2003 to Mr. Hammerle of LARC and in my fax of June 7, 2003 to 
you, I have no way of determining exactly which claims the X-38 project may have infringed unless 
NASA makes a full and complete disclosure to me of that project. I also have no way of determining if 
NASA has (or has had) other projects that also infringe on my patent unless NASA makes a full and 
complete disclosure of those projects as well. 

Therefore, in order to answer your question, I must request that NASA make a full and complete 
disclosure to me of the X-38 project as well as any other current or past projects that may infringe on my 
patent. 

If this information requires a security clearance (I have none) I suggest you start the required security 
investigation immediately. If there is further information that you require in this regard fee! free to contact 
me. 
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(2) The identification of all procurements known to the claimant or patent owner 
which involve the alleged infringing item or process, including the identity of 
the vendor or contractor and the Government procuring activity. 

As f stated in my fax to you of June 7, 2003, I became aware that NASA was using synthetic vision in 
the X-38 project in the January 2003 issue of NASA Tech Briefs, page 40, "Virtual Cockpit Window" 
for a Windowless Aerospacecraft. The article is available at: 
http://www.nasatech.com/Briefs/Jan03/MSC23096.html 

This led me to Rapid Imaging Software, Inc. and their press release 
( http://www.landform.com/pages/PressReleases.htm ) which states: 

"On December 13th, 2001, Astronaut Ken Ham successfully flew the X-38 from a remote cockpit 
using LandForm VisualFlight as his primary situation awareness display in a flight test at 
Edwards Air Force Base, California. This simulates conditions of a real flight for the windowless 
spacecraft, which will eventually become NASA's Crew Return Vehicle for the ISS. We believe 
that this is the first test of a hybrid synthetic vision system which combines nose camera video 
with a LandForm synthetic vision display. Described by astronauts as 'the best seat in the 
house', the system will ultimately make space travel safer by providing situation awareness 
during the landing phase of flight." 

The RIS press release provided a link to an article in Aviation Week & Space Technology: 
http://www.aviationnow.com/avnow/news/channel space.jsp?view=storv&id=news/sx38121 1 .xml 

As a result of more searching I discovered a link to a Johnson Space Center SBIR Phase II award to 
Rapid Imaging Systems at http://sbir.gsfc.nasa.gov/SBIR/successes/ss/9-058text.html . 

It includes a particularly relevant paragraph: 

The Advanced Flight Visualization Toolkit (VisualFlight™) project is developing a suite of 
virtual reality immersive telepresence software tools which combine the real-time flight 
simulation abilities with the data density of a Geographic Information System (GIS). This 
technology is used for virtual reality training of crews, analysis of flight test data, and as an on- 
board immersive situation display. It will also find application as a virtual cockpit, and in 
teleooeration of remotely piloted vehicles . 

The emphasis on virtual reality immersive telepresence and teleoperation of remotely piloted vehicles is 
mine. 



A search of the SBIR archive shows the following entries. 

For 2001 Phase I: 
Rapid Imaging Software, Inc. 
1318 Ridgecrest Place S.E. 
Albuquerque, NM 87108-5136 
Mike Abernathy (505) 265-7020 
01 H6.02-8715 JSC 

Integrated Video for Synthetic Vision Systems 
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For 2001 Phase II: 
Rapid Imaging Software, Inc. 
1318Ridgecrest Place S.E. 
Albuquerque , NM 87108-5136 
Carolyn Galceran ( 505 ) 265 - 7020 
01-2-H6.02-8715 JSC 

Since my sources of information are limited to those available to the public (magazines such as Aviation 
Week & Space Technology as well as whatever I can find on the Internet) I have no way of knowing if 
there are other procurements, vendors, contractors, and Government procuring activity related to Claim 
1-222. 

I believe that NASA is in a better position to know what it is (or has been) working on than I am. 



(3) A detailed identification of the accused articles or processes, particularly where the 
article or process relates to a component or subcomponent of the item procured, 
an element by element comparison of the representative claims with the accused 
article or process. If available, this identification should include documentation 
and drawings to illustrate the accused article or process in suitable detail to enable 
verification of the infringement comparison. 

I believe I have answered this in section (2) as much as I am able to without NASA's cooperation. 



(4) The names and addresses of all past and present licenses under the patent(s), and 
copies of all license agreements and releases involving the patent. 

There are no past licenses for this patent, and as of this date there are no present licenses for this 
patent. Naturally, I reserve the right to license this patent in the future as I see fit. 



(5) A brief description of all litigation in which the patent(s) has been or is now 
involved, and the present status thereof. 

There has been no past litigation involving this patent, and as of this date there is no present litigation 
regarding this patent. 
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(6) A list of all persons to whom notices of infringement have been sent, including 

all departments and agencies of the Government, and a statement of the ultimate 
disposition of each. 

As of this date NASA is the only agency or department of the Government against which I have filed a 
claim. 



5/11/03- sent email to comments(3)ha.nasa.aov 

I believe that NASA may have infringed on one or more of my U. S. Patents. 
How do I file a claim and whom do I contact? 



5/1 1/03 - Received reply: 

Date: Sun, 11 May 2003 17:48:46 -0400 (EDT) 

From: "PAO Comments" <comments@bolg.public.hq.nasa.gov> 

Message-ID: <200305112148. h4BLmkh J011314@bolg. public. hq.nasa.gov> 

To: <jm@jmargolin.com> 

Subject: Thank you for your email. 

Thank you for your message to the NASA Home Page. The Internet 
Service Group will attempt to answer all e-mail regarding the site, 
but cannot guarantee a response by a particular time. The group 
will not be able to answer general inquiries regarding NASA, 
which should instead be sent to Dublic-inauihes(8)ha. nasa. gov 



5/1 1/03 - Sent email to <public-inquiries@hq.nasa.gov> 

/ believe that NASA may have infringed on one or more of my U.S. Patents. 
How do I file a claim and whom do I contact? 

Jed Margolin 
As far as I can tell I did not receive a response. 



5/12/03 - Sent email to i.c.midqett(a)larc.nasa.qov (found on Web site) 

/ believe that NASA may have infringed on one or more of my U.S. Patents 
How do I file a claim and whom do I contact? 
(Oris my only recourse to sue in Federal Court?) 

Jed Margolin 
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5/12/03 - Received reply: 
Mr. Margolin, 

Thank you for contacting NASA with your concerns. I have referred this 
matter to the Patent Counsel Office, and they will be contacting you to 
work with you on this issue. 

Best wishes, 
Jesse Midgett 

5/12/03 - Given my experience with trying to contact Government officials via email (or mail, or fax) 
I hadn't waited for the reply from J. Midgett. I had found the web site for the LARC (NASA Langley) 
Patent Counsel Office, and called up. I was connected to Kurt Hammerle and we had a nice talk. I 
sent him an email the next day (May 13, 2003). 

I received a phone call from Barry Gibbens (757-864-7141) who, apparently, was calling because of 
my email to to J.C.Midgett and hadn't seen the email I sent to K. Hammerle. (I explained to him what 
I had done.) We had a nice talk. He said he had already sent me a letter. 

I received his letter and sent a reply on May 18, 2003 (USPS), adding to the email I had sent K. 
Hammerle. 

Thursday, June 5, 2003 - Received message from B. Gibbens, asking me to call him because I 
should contact Alan Kennedy at NASA Headquarters (202-358-2065). 

Friday, June 6, 2003 - 1 called B. Gibbens. Then I called A. Kennedy but he was out. 

Saturday June 7 2003 - Sent a fax to A. Kennedy. The first number I tried (202-358-4341 ) only 
accepted 4 pages (out of 13). I tried a few times. Then I tried 202-358-2741 . It turned out that 4341 
was the correct number and that 2741 was another group. As a result, A. Kennedy initially only got 4 
pages. 

Monday, June 9, 2003 - Received message from A. Kennedy and called him back. 

He had not gotten the fax so he went and found it. I learned the next day that he had only gotten 4 
pages. 

We had a "free and frank" discussion, I stressed that I wanted to resolve it in a friendly manner and 
that I preferred to have NASA buy the patent for the Government. 

Tuesday, June 10, 2003 - Received a message from A. Kennedy and called him back. 

He said that his Manager has turned down my request that NASA consider a license proffer and has 
decided to handle it as a Claim, and that the investigation would take 3-6 months. 
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However, NASA is not the only agency or department of the Government I have contacted. 



7/5/1999 Email to: lbirckelbaw@darpa.mil 

Dr. Birckelbaw, Project Manager for the UCAV contract awarded to Boeing. 

Introduced myself and asked if DARPA was interested in my patent. 
Response: none 



7/26/1999 USPS Mail to: 
Dr. Larry Birckelbaw 
Program Manager, Aerospace Systems 
DARPA Tactical Technology Office 
3701 North Fairfax Drive 
Arlington, VA 22203-1714 

Introduced myself and asked if DARPA was interested in my patent. Enclosed copy of patent. 
Response: none 

Office of the Secretary of Defense (OSD) 
Mr. E.C. "Pete" Aldridge 

Under Secretary of Defense for Acquisition Technology, and Logistics 
U.S. Department of Defense 

Contact Method: Email: webmaster@acq.osd.mil May 3, 2002 and June 6, 2002 

Response: none 



Army - AATD, Fort Eustice, VA. 

Col. Wado Carmona, Commander 

Applied Aviation and Training Directorate (AATD) 

Army Aviation and Missile Command 

Ft. Eustice, VA 



Contact Method: 

Email: Ms. Lauren L. Sebring lsebring@aatd.eustis.army.mil June 1, 2002 

757-878-4828, fax: 757-878-0008 

Phone Call Followup: She suggested I talk to Mr. Jack Tansey 

Mr. Jack Tansey, Business Development 757-878-4105 June 18, 2002 

Email Followup: jtansey@aatd.Eustis.army.mil June 18, 2002 



Air Force Research Laboratory (AFRL) 
Dr. Barbara Wilson 

Contact Method: email (Barbara.Wilson@wpafb.af.mil) July 17, 2002 

Response - none 
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Air Force Research Laboratory (AFRL) 
Dr. R. Earl Good, Director, 
Directed Energy Directorate 
Air Force Research Laboratory 
Kirtland Air Force Base, NM 871 1 7-5776 

Contact Method: Fax (505-846-0423) July 23, 2002 

Response: none 



Department of the Air Force 
Dr. James G. Roche 
Secretary of the Air Force 
Washington, DC 

Contact Method: Fax (703-695-8809) July 28, 2002 

Response: Letter from August 13, 2002 

Lt. General Charles F. Wald 
Deputy Chief of Staff, Air & Space Operations, USAF 



(7) A description of Government employment or military service, if any, by the 
inventor and/or patent owner. 

I have never been employed by the U.S. Government (or any other government). Likewise, I have never 
been in military service (in the United States or elsewhere). In the interests of full disclosure, I worked for 
three summers (1967, 1968, 1969) at the RCA Astro-Electronics Division in Hightstown, NJ . (They had 
a summer job program for students.) 



(8) A list of all Government contracts under which the inventor, patent owner, or 

anyone in privity with him performed work relating to the patented subject matter. 

None. I did this entirely on my own dime. 
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(9) Evidence of title to the patent(s) alleged to be infringed or other right to make the 
claim. 

This appears to be a two-part question. Does the patent belong to Jed Margolin, and am I that Jed 
Margolin? 

Part 1 - If you look at the front page of the 724 patent you will see that it was, indeed, issued to Jed 
Margolin, 3570 Pleasant Echo Dr, San Jose, CA. 

If you contact the U.S. Patent and Trademark Office, Document Services Department (703-308-9726), 
you can order an Abstract of Title to verify that I own the patent. According to 37 CFR 1.12, assignment 
records are also open to public inspection at the United States Patent and Trademark Office. 

Part 2 - If you look up Jed Margolin, 3570 Pleasant Echo Dr., San Jose, CA, in a telephone directory 
you will find assigned to it the telephone number 408-238-4564. 

When you called me on June 9 and June 10, that was the number you called. 

Other than my affirming that I am, indeed, the Jed Margolin in question, I can only suggest that you 
contact my cousin Lenny (oops, I mean Dr. Len Margolin) who is employed by Los Alamos National 
Laboratory, and ask him if he has a cousin Jed who is an engineer and an inventor, and who possesses 
the Margolin gene for being very persistent. (Some say stubborn.) The last time I saw him was in Ann 
Arbor, Michigan, after he had just passed the orals for his doctorate. (He bought me a beer at a place on 
South University.) 



(1 0) A copy of the Patent Office file of the patent, if available, to claimant. 

I do not have a copy of the USPTO's patent file. What I have is my prosecution file which contains, 
among other things, privileged communications between my patent attorney and myself, 

Besides, in our telephone conversation of June 10, you stated that one of the research centers (I believe 
it was LARC) had already ordered the file. 
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(11) Pertinent prior art known to claimant, not contained in the Patent Office file, 
particularly publications and foreign art. 

I have found no relevant prior art. 

However, there is an interesting article in the June 2, 2003 issue of Aviation Week & Space Technology 
on pages 48-51 entitled GA Riding *Highway-in-the-Sky' which describes, among other things, the 
work of Dennis B. Berlinger, lead scientist for flight deck research at the FAA's Civil Aeromedical 
Institute (CAMI) regarding what is called Performance-Controlled Systems. In the Specification of my 
724 patent I call it First Order RPV Flight Control Mode. In Claim 18: 

18. The station of claim 13, wherein said set of remote flight controls are configured to 
allow inputting absolute pitch and roll angles instead of pitch and roll rates. 

An Internet search turned up Mr. Beringer's report Applying Performance-Controlled Systems, Fuzzy 
Logic, and Fly-By-Wire Controls to General Aviation as DOT/FAA/AM-02/7. 

I am pleased that Mr. Beringer's May 2002 study confirms the value of Performance-Controlled Systems 
in piloted aircraft and I believe that teaching it in my 724 patent (filed January 19, 1999) gave an 
additional novel and useful aspect to my invention. 

(The article also describes the Synthetic Vision system used in the FAA's Capstone program.) 



If you have any further questions, please contact me. 



Sincerely yours, 



Jed Margolin 



Enclosed: Response from General Wald 
AWST article 
Beringer Report 
U.S. Patent 5,904,724 
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HEADQUARTERS UNITED STATES AIR FORCE 
WASHINGTON, DC 



DEPARTMENT OF THE AIR FORCE 



13 Aug 02 



HQ USAF/XO 

1630 Air Force Pentagon 

Washington, DC 20330-1630 

Mr. Jed Margolin 
3570 Pleasant Echo Dr. 
San Jose, CA 95148-1916 

Dear Mr. Margolin 

On behalf of Secretary Roche, thank you for providing your ideas on ways to improve 
UAV control technology. As you know, we are now operating the Global Hawk and Predator 
systems in reconnaissance roles, and envision expanding unmanned aircraft applications into the 
weapons delivery mission area with the UCAV and the Predator/Predator B aircraft. Certainly 
we see a growing role for UAVs in the Air Force as technology advances and we gain experience 
in their operation. The improved control methods you have patented may well play a part in 
future UAV design. I suggest that you present these concepts to the various UAV manufacturers 
who are in the business of designing systems to meet our operational requirements. They can 
offer the best assessment on the overall feasibility of integrating your technology. I suggest a 
similar approach regarding your patented laser techniques. 

Again, thank you for taking the time to offer these suggestions. I admire your ingenuity, 
and appreciate your desire to help us improve our national defense capabilities. 



Sincerely 




CHARLES F. WALD, Lt Gen, USAF 
Deputy Chief of Staff 
Air & Space Operations 



cc: 



SAF/AQ 
AF/XOR 
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United States Patent 

Margolin 
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US005904724A 
[11] Patent Number: 
[45] Date of Patent: 



5,904,724 
May 18, 1999 



[54] METHOD AND APPARATUS FOR 

REMOTELY PILOTING AN AIRCRAFT 

[76] Inventor: Jed Margolin, 3570 Pleasant Echo, San 
Jose, Calif. 95148 

[21] Appl. No.: 08/587,731 
[22] Filed: Jan. 19, 1996 

[51] Int. CL 6 G06F 165/00; H04N 7/18 

[52] U.S. CI - 701/120; 701/2; 701/24; 
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[57] ABSTRACT 

A method and apparatus that allows a remote aircraft to be 
controlled by a remotely located pilot who is presented with 
a synthesized three-dimensional projected view representing 
the environment around the remote aircraft. According to 
one aspect of the invention, a remote aircraft transmits its 
three-dimensional position and orientation to a remote pilot 
station. The remote pilot station applies this information to 
a digital database containing a three dimensional description 
of the environment around the remote aircraft to present the 
remote pilot with a three dimensional projected view of this 
environment. The remote pilot reacts to this view and 
interacts with the pilot controls, whose signals are transmit- 
ted back to the remote aircraft. In addition, the system 
compensates for the communications delay between the 
remote aircraft and the remote pilot station by controlling 
the sensitivity of the pilot controls. 

20 Claims, 7 Drawing Sheets 
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GA Riding 'Highway-in-the-Sky' 

General aviation sector reaps the benefits of research 
originally conducted for military, commercial transport cockpits 

BRUCE D. NORDWALiy WASHINGTON and OKLAHOMA CITY 



J0^i eneral aviation aircraft are fi- 
W nally catching up with some of 
n mm the advances found in the lat- 
I est commercial transports and 
^ i BgP military cockpits, and in one 
particular sphere — display innova- 
tions — GA is actually taking the lead. 

Researchers in industries and uni- 
versities around the world have been 
pursuing a more intuitive guidance dis- 
play for pilots for years. In general, this 
elusive presentation is referred to as 
highway-in-the-sky (HITS) (AW&ST 
Apr. 20, 1998, p. 58). In a twist that may 
foreshadow future advances, it was a 
general aviation aircraft that received 
the FAA's first certification of HITS 
technology for navigation guidance. 
Instead of following course deviation 



indicators and altimeters, a pilot using 
this HITS presentation flies through a 
series of 3D boxes on a multifunction 
display. By maneuvering through the 
400 X 320-ft. boxes spaced at 2,000-ft 

Flying through "boxes in the sky" keeps 
pilots on course and altitude during a simu- 
lated curved instrument approach down the 
mountainous Castineau Channel to Juneau, 
Alaska. 

intervals along the planned GPS route 
of flight, the pilot keeps the aircraft on 
course and altitude, which is particularly 
helpful for a descending, curved instru- 
ment approach. 

L.A.B. Flying Service's Piper Seneca 
made the first commercial revenue flight 
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using HITS in Juneau, Alaska, on Mar. 
31. It followed an optimized area navi- 
gation (RNAV) route through airspace 
that would be inaccessible with con- 
ventional avionics. 

The system was built by Chelton 
Flight Systems as part of the second 

www.AviationNow.com/awst 
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phase of the imaginative Capstone pro- 
gram, an FAA industry/academic part- 
nership in Alaska. The cockpit employs 
a Chelton FlightLogic electronic flight 
information system-synthetic vision 
(EFIS-SV) using two glass displays, one 
for primary flight guidance and one for 
navigation. 

The big innovation is the use of syn- 
thetic vision symbology to present in- 
formation to pilots. The initial EFTS sys- 
tems digitally replicated the rudimentary 
attitude and flight-director symbols of 
electro-mechanical instruments from an 
earlier era. Now, in addition to the flight 
path, pilots see a real-time 3D view of 
the terrain and obstacles on the primary 
flight display. These are complemented 
by a moving map on the navigation dis- 
play and by aural terrain warnings. 

Among the other "firsts" claimed by 
Capstone Phase II on the Juneau flight 
were the use of forward-looking 3D ter- 
rain and HUD symbology on a certified 
primary flight display, and commercial 



Automatic De- 
pendent Surveil- 
lance-Broadcast 
(ADS-B) equip- 
ment (AW&ST 
Sept. 18, 2000, 
p. 68). With GPS 
as the enabling 
technology, that 
phase indicated 
that a low-cost sys- 
tem could give 
bush pilots many of the 
safety benefits long-stan- 
dard for commercial jet 
transports. The emphasis 
was on reducing con- 
trolled flight into terrain 
accidents for these pilots, 
who usually operate out of 
the range of navigation 
aids or radar help from 
ATC. Phase II with HITS 
and synthetic vision greatly 
expands those capabilities. 




The navigation display 
shows GPS WAAS position 
and an approach not 
possible with conventional 
navigation aids due to a 20- 
30-deg. turn after (he 
GASTN waypoint to align 
with the runway. 



CAM! tested a four-axis skte-arm controller in a simulator as a replacement 
for stick and throttle in a fly-by-wire performance control system. 



use of the GPS wide-area augmentation 
system (WAAS). 

Capstone has equipped three aircraft 
in Alaska with the Chelton Flight Sys- 
tems' cockpit, and plans to outfit every 
commercial operator in SE Alaska with- 
in the next 18 months. The contract for 
125 aircraft could expand to up to 200, 
according to Gordon Pratt, Chelton's 
president. The FAA is providing the 
equipment at no charge in Alaska to any 
commuter and on-demand (FAA Part 
135) operator of fixed-wing aircraft or 



The next major safety 
enhancement for GA air- 
craft could come from 
"performance control," ac- 
cording to Dennis B. Beringer, lead sci- 
entist for flight deck research at the FAAs 
Civil Aeromedical Institute (CAMI) in 
Oklahoma City. While known more for 
assisting FAAs Aircraft Certification Ser- 
vice and Flight Standards in defining 
requirements for both aircraft and pilots, 
CAMI is also an active partner in human 
factors research to improve cockpits. 



with performance control, 

non-pilots could learn to fiy 

a simulator in 15 min. 



helicopters. A supplemental type cer- 
tificate for helicopters was scheduled to 
be delivered on May 31. An additional 
10 aircraft are being outfitted in the 
contiguous U.S., Pratt said, but at the 
expense of aircraft owners. 

The first phase of the Capstone Pro- 
gram started as a demonstration that 
equipped a number of commuter and 
air taxi aircraft in the Yukon-Kuskok- 
wim River delta area with a low-cost 
GPS, a terrain database, data link and 



The performance-control concept was 
introduced in the 1970s, before elec- 
tronics were sufficiently advanced for 
implementation. Beringer said that now 
some of the fly-by-wire military and 
commercial aircraft use what could be 
legitimately called performance-control 
logic, which not only make aircraft eas- 
ier to fly, but can also add flight enve- 
lope protection. 

With conventional flight controls, a 
pilot has direct command of the aero- 
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dynamic surfaces. 
With performance 
control, his move- 
ments would be 
transmitted via a 
fuzzy-logic con- 
troller to a flight 
management sys- 
tem or an auto pi- 
lot that would 
guide the aircraft 
to carry out the 
desired performance goal. 
But unlike a simple au- 
topilot, which directs a 
change in heading at a lim- 
ited rate of turn, perform- 
ance-control logic changes 
control laws so that a pilot 
commands the rate of turn 
and bank, and rate of 
climb or descent. It sim- 
plifies command of more 
complicated maneuvers, 
and is a compromise be- 
tween automated maneu- 
vering and manual flight 
control, Beringer said. 
Safety is further enhanced 
using a self-centering 
(spring-loaded) side stick 
which returns to the cen- 
tered position when the pi- 
lot relaxes pressure, thus bringing the 
aircraft to straight and level flight. 

The reduced number of control move- 
ments is one reason flying is easier. 
Going into a turn with conventional con- 
trols, the pilot has to initiate the roll, 
and then neutralize the ailerons when 
he achieves the desired bank angle. But 
with performance controls, one move- 
ment establishes the desired bank 
angle/turn rate. One downside to per- 
formance control with envelope pro- 
tection is the inability to do aerobatics, 
such as an aileron roll or loop, Beringer 
said. 

In the four-axis side-arm controller 
(above), rotating the wrist governs the 
rate of turn, flexing the wrist vertically 
directs the rate of climb or descent, and 
fore and aft movement varies the air- 
speed. Interest in performance controls 
was renewed with NASA's Agate (Ad- 
vanced General Aviation Transport Ex- 
periments) program, which was con- 
cerned with simplifying the flight task 
and reducing ab initio training require- 
ments. Agate has also been a strong 
supporter of HITS. 

Researchers had previously found that 
with performance control, non-pilots 
could learn to fly a simulator in 15 min. 
Beringer tested the system in a simula- 
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tor configured as a Piper Malibu at 
CAMI. It used HITS displays and a four- 
axis side-arm controller. Twenty-four in- 
dividuals with varying flight experience 
participated: six high-flight-time pilots; 
six low-flight-time pilots; six student pi- 
lots, and six non-pi!ots. Each flight in- 
volved a takeoff into instrument con- 
ditions, a continuous climb while turning 
downwind, a turn to intercept the in- 
strument landing system glidepath, and 
a descent to landing. Flights were di- 
vided between use of a conventional 
yoke and the side-arm controller. 

The findings were consistent. The air- 
craft was more stable and had less vari- 
ations in course and altitude using per- 
formance control than with conventional 
controls. Although experienced pilots 



always outperformed less-experienced 
individuals, with either system, all agreed 
the effort required was nearly halved. 

Performance control is not apt to be 
seen in Piper Cubs, but perhaps in Beech 
Bonanzas and Piper Malibus. A lot of 
them already have two- or three-axis au- 
topilots, so a significant capability could 
be achieved by rigging a side-stick con- 
trol to the autopilot, Beringer said. 

But two large problems must be over- 
come for performance controls to ap- 
pear in the next generation of GA air- 
craft. The first is cost. Affordable and 
certifiable computer controls and ser- 
vos would have to drop to a level com- 
petitive with more conventional systems. 

Second, a fly-by-wire debate must be 
resolved. Could an affordable system be 
built with sufficient reliability using 
triple- or quad-redundancy, or would a 
costly manual-reversion be required? A 
mechanical backup would add cost for 
installation and for training pilots to op- 
erate the two systems. 

Complicating that issue is the question 
of the level of reliability required. The 
FAAs current standard for a flight-criti- 
cal system is a failure rate of 10"'. While 
this is a standard for NASA it might not 
be reasonable for general aviation air- 
craft, Beringer points out that the failure 
rate for humans is about 10 J . 9 

ww.AviationNow.com/awst AVIATION WEEK & SPACE TECHNOLOGY/JUNE 2, 2003 51 




A15 



Case 3:09-cv-00421 -LRH-VPC Document 1-1 Filed 07/31/09 Page 1 6 of 1 28 



Exhibit 

2 



A16 



Case 3:09-cv-00421 -LRH-VPC Document 1-1 



Filed 07/31/09 Page 17 of 128 

Page 1 of 1 



Jed Margolin 



From: "Jed Margolin" <jm@jmargolin.com> 

To: <nasafoia@nasa.gov> 

Sent: Saturday, June 28, 2008 7:05 PM 

Attach: jm_nasa.pdf 

Subject: FOIA Request 

This request is made pursuant to the Freedom of Information Act. 

I would like all documents related to the Administrative Claim of Jed Margolin for Infringement of U.S. 
Patent Nos. 5,566,073 and 5,904,724; NASA Case No. 1-222. 

I am attaching a letter dated June 11, 2003 from Alan Kennedy, Director, Infringement Division, Office of the 
Associate General Counsel as file jm_nasa.pdf. I provided the information requested, it was received by Mr. 
Kennedy, and thereafter Mr. Kennedy refused to respond to my attempts to find out the results of the 
investigation. 

I believe NASA has had enough time to have completed its investigation by now. 

Jed Margolin 
1981 Empire Rd. 
Reno,NV 89521-7430 
775-847-7845 
www.jmargolin.com 
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National Aeronautics and Space Administration 

Headquarters 

Washington, DC 20546-0001 




August 5, 2008 



Reply to Attn of: 



Office of the General Counsel 



Mr. Jed Margolin 
1981 Empire Road 
Reno, NV 89521-7430 



Administrative Claim of Jed Margolin for Infringement of U.S. Patent 
Nos. 5,566,073 and 5,904,724; NASA Case No. 1-222. 



Dear Mr. Margolin, 

We are in receipt of the Freedom of Information Act Request (FOIA) conveyed to us by email dated 
June 30, 2008 in which you request copies of ail documentation relating to your administrative claim 
of infringement of U.S. Patent Nos. 5,566,073 and 5,904,724. 

We regret the delay in processing your claim and assure you that we are now undertaking measures 
to provide a resolution of your claim as soon as possible. Unfortunately. Mr. Alan Kennedy retired 
from NASA earlier this year and the action on your claim was not conveyed to management in a 
timely manner. In addition the local attorney responsible for review of your claim also departed 
from NASA. We are now cognizant of the importance of proceeding with a review of the claim and 
will contact you when we have reached a decision. 

As to your FOIA request, as the investigation of your claim is ongoing, we kindly request that you 
allow us a 90 day extension to answer this request. Within that time period we should be able to 
obtain a better picture of our position vis-a-vis your claim and the request for documents may no 
longer be required. 

We should inform you that we have received a separate communication from a company Optima 
Technology Group, claiming to have been assigned both of the patents in question. You informed 
me telephonically that this is the case; however, we have no record of any assignment of your patents 
to this firm and will need confirmation through appropriate attested documents delivered to the 
agency in order to recognize any claim of ownership by a party other than the inventor. 

Thank you for your patience in this matter. Please contact the undersigned at (202) 358-0632 or 
email Jan .McNu tt@nasa.gov if you have any additional questions or comments. 




Sincerely, 



Attorney-Advisor 
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Jed Margolin 
Phone: 775-847-7845 



1981 Empire Rd. 
jm@jmargolin.com 



Reno,NV 89521-7430 
August 8, 2008 



Mr. Jan S. McNutt 
Office of the General Counsel 
NASA Headquarters 
Washington, DC 20546-0001 

Re: FOIA Request (FOIA HQ 08-270) regarding NASA Case No. 1-222 



Dear Mr. McNutt, 



As we discussed in our recent telephone conversations, my FOIA Request is entirely separate 
from NASA Claim Case 1-222. The patents involved in the claim are now owned by Optima 
Technology Group, Inc. I trust that Optima Technology Group has now provided you with the 
documentation you requested in order to establish their ownership of the Patents. 

I will agree to the 90 day extension you have requested for NASA to respond to my FOIA 
Request (HQ 08-270) if NASA acknowledges that my FOIA request is entirely separate from 
Optima Technology Group's Claim Case No. 1-222. 



Sincerely yours, 




Jed Margolin 
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Jed Margolin 
Phone: 775-847-7845 



1981 Empire Rd. 



Reno, NV 89521-7430 
April 27, 2009 



Mr. Christopher J. Scolese, 
Acting Administrator, NASA 
300 E Street, SW 
Washington, DC 20546 
(202) 358-2810 (Fax) 



Dear Sir, 

I sent you a letter by certified mail on April 6, 2009. According to USPS it has not been delivered. 
USPS has several theories: 

1. They lost it; 

2. NASA refused to accept delivery; 

3. Something happened to it when it was sent to New Jersey to be irradiated. 
I am appending the letter to this fax. 

The letter asks you to confirm that I have exhausted all of the administrative remedies that NASA 
has to offer in my attempt to get NASA to comply with the Freedom of Information Act. Since it 
took me an hour this morning just to get a fax number for you -I was misdirected all around NASA- 
the answer is obviously, "Yes." 

When I file suit against NASA in the U.S. District Court For the District of Nevada I had planned to 
mail the Complaint to you. Since it does not seem possible to mail anything to NASA with any hope 
of success, will you allow me to email or fax the Complaint to you and will you waive Service? 

If you refuse, I will have to pay a process server to serve you. Then I will amend my Complaint to 
ask the Court to assess costs and punitive damages against NASA. 



Sincerely yours, 



Jed Margolin 



Cc: Senator Harry Reid 
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Phone: 775-847-7845 



Jed Margolin 



1981 Empire Rd. 



Reno,NV 89521-7430 
April 6, 2009 



Mr. Christopher J. Scolese, 
Acting Administrator, 
NASA 

300 E. Street, SW 
Washington, DC 20546 

Dear Sir, 

NASA has been acting in bad faith toward me for the past almost-6 years. 

I am the named inventor on U.S. Patent 5,904,724 Method and apparatus for remotely piloting an 
aircraft issued May 1 8, 1999. This patent teaches the use of (what is now called) synthetic vision for 
controlling a UAV. 

1 contacted NASA in May 2003 after I became aware that NASA had used synthetic vision in the X-38 
project. Because the use of synthetic vision for controlling a UAV can be used to the detriment of this 
country by unfriendly entities I wanted a friendly conversation because I thought NASA should buy the 
patent in order to control the technology 

In June 2003 I was turned over to Mr. Alan Kennedy in the Office of the General Counsel. This is what I 
recorded in my Contact Log: 

Summary: He basically said that what most independent inventors have is junk and that since I am an 
independent inventor what I have is probably junk. If NASA evaluates it as a license proffer it will give it 
a pro forma rejection and I will file a claim anyway, so the same people who rejected it as a proffer will 
reject it as a claim, but in the process will have had to do more work, so to save them some work they 
will ignore the proffer and handle it as a claim. 

So, I filed a claim, completely answering all the questions on NASA's claim form. Then Mr. Kennedy 
informed me that NASA would conduct an investigation (expected to last 3-6 months) and that the purpose 
of the investigation would be to find prior art to invalidate the patent. 

After six months I did not hear from NASA so I called Mr. Kennedy. He said: 

1 . The investigation had not been done. 

2. NASA had a Research Exemption for using the patent. [Not true. See Madey v. Duke 307 F.3d 1 35 1 
(Fed. Cir. 2002)] 

3. "The X-38 never flew." I informed him of the video on NASA's web site showing the X-38 flying. 
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4. The Statute of Limitations gives NASA 6 years to respond to my claim. (Wrong, it gives me 6 years 
to take NASA to Federal Claims Court.) 

5. It would cost me more to sue NASA in Federal Claims Court than I could hope to recover from 
NASA. 

After that, Mr. Kennedy refused to talk to me or respond to my letters. Then, various things came up and I 
was unable to pursue my claim against NASA. 

Subsequently, I assigned the patent to Optima Technology Group, which has inherited the claim. 

However, I still wanted to know what came up during the investigation so, on July 1, 2008 I filed a FOIA 
request. It was assigned FOIA HQ 08-270. 

For some reason it was turned over to Mr. Jan McNut in the Office of the General Counsel. 
His response is attached as Reference 1. 

On August 5, 2008 Mr. McNut asked me to give NASA a 90-day extension to my FOIA request. I agreed. 

In January, 2009 I received a letter from Mr. McNut who sent me back to the FOIA Office (See Reference 2), 
who wanted me to start over from scratch. Ms. Kelly Robinson then explained that she was currently 
working on FOIA requests filed two years before. 

I told her that NASA did not get a do-over. 

In the interests of brevity: 

1 . I talked to Ms. Robinson on March 18, 2009. She said she was sending me the results of the FOIA 
search, but there was some material she would not send me before it was internal Agency 
communications. 

2. That was almost three weeks ago I have not received anything from NASA. 

Therefore, Mr. Scolese, please confirm that I Have Exhausted All the Administrative Remedies that 
NASA Has to Offer. I need you to do this so I can bring suit against NASA in Federal Circuit Court. 

If you fail to respond to this letter within ten days I will assume the answer is "Yes." 

And I will note your failure to respond in my upcoming article, "How NASA Defrauds Independent 
Inventors." (That was not the title when this process started.) I will be sending the article to the various 
House and Senate oversight committees. 

Sincerely yours, 



Jed Margolin 
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Jed Margolin 



From: 

To: 

Sent: 

Attach: 

Subject: 



"HQ-FOIA" <hq-foia@nasa.gov> 
<jm@jmargolin.com> 
Monday, May 18, 2009 12:30 PM 
2008-270.pdf; 08-270.DOC 
FOIA 2008-270 



FOIA 08-270 



May 14, 2009 



Mr. Jed Margolin 
1981 Empire Road 
Reno, NV 89521-7430 

jm@ jmar golin.com 

Dear Mr. Margolin: 

This is in response to your request received on June 30, 2008, pursuant to the Freedom of 
Information Act (FOIA) for documents related to the Administrative Claim of Jed Margolin for 
infringement of U.S. Patent Nos. 5,566, 073 and 5,904,724; NASA Case No. 1-222. 

The NASA Headquarters Office of the General Counsel conducted a search and from that search 
provided the enclosed documents responsive to your request. 

It has been determined that portions of the records found responsive to your request contain 
information which is exempt from disclosure under the deliberative process privilege of Exemption 5. 
This privilege covers advisory opinions, recommendations, and deliberations, which are part of the 
government decision-making process, 5. U.S.C.§552(b)(5). 

You may appeal this initial determination to the NASA Administrator. Your appeal must (1) be 
addressed to the Administrator, National Aeronautics and Space Administration, Washington, DC 
20546, (2) be clearly identified on the envelope and in the letter as an "Appeal under the Freedom of 
Information Act", (3) include a copy of the request for the agency record and a copy of this initial 
adverse determination, (4) to the extent possible, state the reasons why you believe this initial 
determination should be reversed, and (5) be sent to the Administrator within thirty (30) calendar days 
of the receipt of this initial determination. 

I apologize for the delay in processing your request. I appreciate your patience. 

Sincerely, 

Original Signed 

Kellie N. Robinson 

FOIA Public Liaison Officer 

Headquarters 

NASA 

300 E Street, SW 
Washington, DC 20546 

Enclosures 
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From: McConnell, Stephen (HQ NB000) [mailto:stephen.mcconnell-l@nasa.gov] 

Sent; Tuesday, July 01, 2008 8:45 AM 

To: foia@hq.nasa.gov 

Cc: Robinson, Kellie N. (HQ-NB000) 

Subject: FW: FOIA Request 



From: Jed Margolin [mailto:jm@jmargolin.com] 
Sent: Saturday, June 28, 2008 10:06 PM 
To: nasafoia@nasa.gov 
Subject: FOIA Request 

This request is made pursuant to the Freedom of Information Act. 

I would like all documents related to the Administrative Claim of Jed Margolin for Infringement of U.S. 
Patent Nos. 5,566,073 and 5,904,724; NASA Case No. 1-222. 

I am attaching a letter dated June 11, 2003 from Alan Kennedy, Director, Infringement Division, Office of the 
Associate General Counsel as file jm_nasa.pdf. I provided the information requested, it was received by Mr. 
Kennedy, and thereafter Mr. Kennedy refused to respond to my attempts to find out the results of the 
investigation. 

I believe NASA has had enough time to have completed its investigation by now. 

Jed Margolin 
1981 Empire Rd. 
Reno.NV 89521-7430 
775-847-7845 

www .jmargolin .com 
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National Aeronautics and Space Administration 

Headquarters 

Washington, DC 20546-0001 

March 19, 2009 

Reply to Attn of Office of the General Counsel CERTIFIED MAIL 



Dr. Robert Adams, CEO 
Optima Technology Group 
1981 Empire Road 
Reno,NV 89521 



RE: Administrative Claim for Infringement of US Patent No. 5,904,724; 
NASA Case No. 1-222 



Dear Dr. Adams: 

This letter concerns the above-identified administrative claim for patent infringement. 

NASA received the initial notification of this claim in an email dated May 12, 2003, from 
Mr. Jed Margolin addressed to attorneys at the NASA Langley Research Center claiming 
that "NASA may have used one or more of [Mr. Margolin's] patents in connection with the 
X-38 project and may be using one or more of my patents in other projects using Synthetic 
Vision". Mr. Margolin identified two patents that he believed NASA may be infringing; the 
subject patent and Patent No. 5,566,073. On June 7, 2003, Mr. Margolin submitted his 
claim by fax to the NASA HQ attorney, Mr. Alan Kennedy. Mr. Kennedy responded by 
letter dated June 11, 2003 acknowledging the administrative claim and requesting that Mr. 
Margolin give a more detailed breakdown of the exact articles or processes that constitute 
the claim. Mr. Margolin responded by letter dated June 17, 2003, withdrawing his claim 
with regard to U.S. Patent No. 5,566,073, leaving the remaining claim for the subject patent. 
NASA is aware of the long pendency of this matter and we regret the delay. 

On July 14, 2008 Optima Technology Group sent a letter addressed to Mr. Kennedy stating 
that they were the owners of the Jed Margolin patents due to an assignment and requesting 
that NASA now license the technology of the subject patent. With an email dated August 6, 
2008 from Optima, NASA received a copy of a Patent Assignment, dated July 20, 2004, 
executed by Jed Margolin, the sole inventor on the subject patent, by which the entire right, 
title and interest in the patent has been assigned to Optima Technology Group, Inc. We 
previously noted in a letter dated August 20, 2008 from Mr. Jan McNutt of our office 
addressed to you that NASA believes there are certain irregularities surrounding this and 
collateral assignment documents associated with the subject patent. However, NASA will at 
this time forestall a detailed consideration of that issue. Instead, we will assume your bona 
fides in asserting that you are the legitimate owner of the subject patent and communicate 
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our findings directly with you. To the extent that Mr. Margolin has any interest in this 
matter, formally or informally, we will leave it up to you whether or not to communicate 
with him. 

In light of the prior claim by Mr. Margolin, we consider your license proffer as an 
administrative claim of patent infringement. We turn now to the substance of your claim. 
In response to your initial letter dated July 14, 2008, Mr. McNutt's August 20, 2008 letter 
posed a number of questions, the purpose of which was to enable NASA to fully evaluate 
the details of your claim. Your organization failed to respond to these questions and, 
further, advanced the position that this matter does not involve a new claim (Adams letter to 
McNutt, August 25, 2008). We disagree that this is not a new claim. Nevertheless, NASA 
proceeds - in order to bring closure to this matter - on the basis that this claim centers 
around allegations that infringement arose from activities associated with NASA's X-38 
Program, as advanced by Mr. Margolin. Accordingly, our investigation of this claim 
necessarily reflects the answers previously furnished by Mr. Margolin in response to 
NASA's June 11, 2003 letter to him containing substantially the same set of questions. 

U.S. Patent No. 5,904,724 issued with twenty claims, claims 1 and 13 being the sole 
independent claims. 

In order for an accused device to be found infringing, each and every limitation of the claim 
must be met by the accused device. To support a finding of literal infringement, each 
limitation of the claim must be met by the accused device exactly, any deviation from the 
claim precluding a finding of infringement. See Lantech, Inc. v. Keip Much. Co., 32 F.3d 
542 (Fed. Cir. 1994). If an express claim limitation is absent from an accused product, there 
can be no literal infringement as a matter of law. See Wolverine World Wide, Inc. v. Nike, 
Inc., 38 F.3d 1 192, 1 199 (Fed. Cir. 1994). 

In applying these legal precepts, reproduced below are the relevant portions of claims 1 and 
13. 

Claim 1. A system comprising: 

* * * 

a computer 

* * * 

said computer is. . .for determining a delay time for communicating said flight data between 
said computer and said remotely piloted aircraft, and wherein said computer adjusts the 
sensitivity of said set of one or more remote flight controls based on said delay time, 
(emphasis added.) 

Claim 13. A station for flying a remotely piloted aircraft that is real or simulated comprising: 

* * * 

a computer 

* * * 
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said computer. . . to determine a delay time for communicating. . .flight control information 
between said computer and [a] remotely piloted aircraft, and said computer to adjust the 
sensitivity of [a] set of remote flight controls based on said delay time. . . .(emphasis added.) 

NASA has investigated activities surrounding the X-38 program at its Centers that 
conducted X-38 development efforts and has determined that no infringement has occurred. 
This result is compelled because none of NASA's X-38 implementations utilized a computer 
which is "for determining a delay time for communicating said flight data between said 
computer and said remotely piloted aircraft," as required by claim 1, nor a "computer ... to 
determine a delay time for communicating ... flight control information between said 
computer and [a] remotely piloted aircraft," as required by the limitations of claim 13. 

Given that a computer which measures delay time is lacking from the NASA X-38 
configuration, it follows that the NASA X-38 configuration had no "adjusting of the 
sensitivity of [a] set of one or more remote flight controls based on said delay time", as 
required in claim 1. Similarly, because the NASA X-38 configuration had no "computer to 
determine a delay time for communicating ... flight control information between said 
computer and [a] remotely piloted aircraft, the configuration also had no adjusting of "the 
sensitivity of [a] set of remote flight controls based on said delay time", as called for by 
claim 13. 

For at least the above-explained exemplary reasons, claims 1 and 13 have not been 
infringed. It is axiomatic that none of the dependent claims may be found infringed unless 
the claims from which they depend have been found to be infringed. Wahpeton Canvas Co. 
v. Frontier, Inc., 870 F.2d 1546 (Fed. Cir. 1989). One who does not infringe an independent 
claim cannot infringe a claim dependent on, and thus containing all the limitations of, that 
claim. Id. Thus, none of claims 2-12 and 14-20 have been infringed. 

NASA's X-38 development efforts ended in 2002. There may also be other features in 
NASA's X-38 development efforts that, upon further analysis, would reveal yet more recited 
claim limitations that are lacking in the NASA configuration related to those efforts. 

We also note as a point of particular significance that the limitations included in claims 1 
and 13 discussed above were added by amendment during the prosecution of the patent 
application. It is clear from an analysis of the patent application file wrapper history that the 
individual prosecuting the application stressed the importance of "the measurement of a 
communication delay in order to adjust the sensitivity of flight controls based on that delay." 
Also noted is the distinguishing arguments that these claims require that there be a 
"computer . . . located in the pilot station" and that "at least one real time measurement of the 
delay and some adjustment is contemplated." (See Applicant's Amendment and Remark, 
February 27, 1998 and Response Under 37 C.F.R, § 1.116, July 6, 1998). Clearly, the Patent 
Office Examiner allowed the application based on these prosecutorial arguments. 

We have completed our investigation regarding the claim of patent infringement of U.S. 
Patent No. 5,904,724 and have determined that there is no patent infringement by, or 
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unauthorized use on behalf of, NASA. The above detailed discussion explains the basis for 
NASA's analysis and decision regarding the subject administrative claim. 

As an aside, during NASA's investigation, numerous pieces of evidence were uncovered 
which would constitute anticipatory prior knowledge and prior art that was never considered 
by the U.S. Patent and Trademark Office during the prosecution of the application which 
matured into Patent No. 5,904,724. In view of the clear finding of lack of infringement of 
this patent, above, NASA has chosen to refrain from a discussion that would demonstrate, in 
addition to non-infringement, supra, invalidity of the subject patent. However, NASA 
reserves the right to introduce such evidence of invalidity in an appropriate venue, should 
the same become necessary. 

This is a FINAL agency action and constitutes a DENIAL of the subject administrative 
claim for patent infringement. 

Pursuant to 35 U.S.C. § 286, the statute of limitations for the filing of an action of patent 
infringement in the United States Court of Federal Claims is no longer tolled. Thus, any 
further appeal of this decision must be made by filing a claim for patent infringement in the 
United States Court of Federal Claims, pursuant to 28 U.S.C. § 1498(a). 




Gary G. Borda 

Agency Counsel for Intellectual Property 
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Jed Margolin 1981 Empire Rd. 

Phone: 775-847-7845 Email: jm@jmargolin.com 



Reno, NV 89521-7430 
June 10, 2009 



Administrator 
NASA Headquarters 
Washington, DC 20546 



Appeal under the Freedom of Information Act to the NASA Response dated May 
14, 2009 and received via email May 18, 2009. 

Jed Margolin FOIA 08-270 Filed: June 28, 2008 



This is an Appeal under the Freedom of Information Act to the NASA Response 
dated May 14, 2009 and received via email May 18, 2009 [Appendix NA1 - NA65] in 
FOIA Request 08-270 filed June 28, 2008 [Appendix NA66\. 

Because NASA's response was sent (and received) on May 18, 2009 this appeal is 

timely. 



In its very tardy response to FOIA Request 08-270 by Jed Margolin ("Margolin") NASA 
withheld documents, citing 5 U.S.C.§552(b(5). 

One of the documents that NASA withheld from Margolin is a letter dated March 19, 
2009 that was sent by Gary G. Borda ("Borda") NASA Agency Counsel for Intellectual 
Property to Optima Technology Group ("OTG"). (This document was given to Margolin 
by OTG.) In this letter Borda denies Claim 1-222 regarding NASA's infringement of U.S. 
Patent 5,904,724 ('724) in the X-38 project. 

Margolin's FOIA 08-270 request to NASA was to produce documents relating to Claim 
1-222 and NASA withheld the most material document so far. 



Sir: 



Summary 
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The Borda letter asserts: 

"... numerous pieces of evidence were uncovered which would constitute 
anticipatory prior knowledge and prior art that was never considered by the U.S. 
Patent and Trademark Office during the prosecution of the application which 
matured into Patent No. 5,904,724." 

And states, "... NASA reserves the right to introduce such evidence of invalidity in an 
appropriate venue, should the same become necessary." 

Circulating the patent report solely within NASA or among other federal agencies is not 
an appropriate venue for NASA to use to have a patent declared invalid. The only 
appropriate venues for NASA to challenge the validity of a U.S. Patent are in the U.S. 
Court of Federal Claims and the Court of Appeals for the Federal Circuit. A Court will 
not accept NASA's word that a patent is invalid due to prior art; NASA would be 
required to produce the evidence. 

Therefore, the exemption under 5 U.S.C.§552(b)(5) does not apply. 

The Borda letter also suggests the existence of other materials and/or documents, 
especially relating to whether NASA risked the X-38 by failing to provide compensation 
for the time delays in the synthetic vision flight control loop. 
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Details 

Most of the documents NASA sent to Requestor Jed Margolin ("Margolin") were 
documents Margolin already had, especially the documents Margolin had himself sent to 
NASA. There were other documents NASA admits to having but refused to provide 
[Appendix NA1]: 

It has been determined that portions of the records found responsive to your 
request contain information which is exempt from disclosure under the 
deliberative process privilege of Exemption 5. This privilege covers advisory 
opinions, recommendations, and deliberations, which are part of the government 
decision-making process, 5. U.S.C.§552(b)(5). 

The reference 5. U.S.C.§552(b)(5) states, referring to Section (a) which requires agencies 
to make information available to the public: 

(b) This section does not apply to matters that are - 



(5) inter-agency or intra-agency memorandums or letters which would not be 
available by law to a party other than an agency in litigation with the agency; 

NASA did not give an estimate of the volume of the documents being withheld, in 
violation of 5 U.S.C.§552(a)(6)(F): 

(F) In denying a request for records, in whole or in part, an agency shall make a 
reasonable effort to estimate the volume of any requested matter the provision of 
which is denied, and shall provide any such estimate to the person making the 
request, unless providing such estimate would harm an interest protected by the 
exemption in subsection (b) pursuant to which the denial is made. 

And, since NASA did not give even a minimal description of the documents being 
withheld, that would probably have been the end of the matter. Without even a minimal 
description of the documents being withheld Margolin would have had no way of 
knowing if NASA was acting properly and in good faith. 
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NASA has a record of acting in bad faith toward Margolin. See: 

1 . Letter from Jed Margolin to Alan Kennedy (NASA Office of the General 
Counsel) dated January 6, 2004 confirming a portion of the telephone conversation 
Margolin had with Kennedy on December 10, 2003 [Appendix NA72] 

2. Fax from Jed Margolin to Acting Administrator Scolese dated April 27, 2009 
detailing NASA's almost-6 years of bad faith shown to Margolin. [Appendix NA73] 

Note that neither document was included in NASA's Response to Margolin's FOIA 
Request, which suggests NASA withheld them in an attempt to avoid embarrassment to 
the Agency and for no other reason. 5 U.S.C.§552(b) does not include "embarrassment to 
the agency" as a reason to withhold documents. 

NASA is still acting in bad faith toward Margolin. 

One of the documents that NASA withheld from Margolin is a letter dated March 19, 
2009 that was sent by Gary G. Borda ("Borda") NASA Agency Counsel for Intellectual 
Property to Optima Technology Group ("OTG"). (This document was given to Margolin 
by OTG.) In this letter Borda denies Claim 1-222 regarding NASA's infringement of U.S. 
Patent 5,904,724 ('724) in the X-38 project. [Appendix NA80] 

Margolin's FOIA 08-270 request to NASA was to produce documents relating to Claim 
1-222 and NASA withheld the most material document so far. 

The Borda letter is so important that it will be reproduced here in its entirety. 



Dear Dr. Adams: 

This letter concerns the above-identified administrative claim for patent infringement. 

NASA received the initial notification of this claim in an email dated May 12, 2003, from 
Mr. Jed Margolin addressed to attorneys at the NASA Langley Research Center claiming 
that "NASA may have used one or more of [Mr. Margolin's] patents in connection with the 
X-38 project and may be using one or more of my patents in other projects using Synthetic 
Vision". Mr. Margolin identified two patents that he believed NASA may be infringing; the 
subject patent and Patent No. 5,566,073. On June 7, 2003, Mr. Margolin submitted his claim 
by fax to the NASA HQ attorney, Mr. Alan Kennedy. Mr. Kennedy responded by letter 
dated June 1 1, 2003 acknowledging the administrative claim and requesting that Mr. 
Margolin give a more detailed breakdown of the exact articles or processes that constitute 
the claim. Mr. Margolin responded by letter dated June 17, 2003, withdrawing his claim 
with regard to U.S. Patent No. 5,566,073, leaving the remaining claim for the subject patent. 
NASA is aware of the long pendency of this matter and we regret the delay. 
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On July 14, 2008 Optima Technology Group sent a letter addressed to Mr. Kennedy stating 
that they were the owners of the Jed Margolin patents due to an assignment and requesting 
that NASA now license the technology of the subject patent. With an email dated August 6, 
2008 from Optima, NASA received a copy of a Patent Assignment, dated July 20, 2004, 
executed by Jed Margolin, the sole inventor on the subject patent, by which the entire right, 
title and interest in the patent has been assigned to Optima Technology Group, Inc. We 
previously noted in a letter dated August 20, 2008 from Mr. Jan McNutt of our office 
addressed to you that NASA believes there are certain irregularities surrounding this and 
collateral assignment documents associated with the subject patent. However, NASA will at 
this time forestall a detailed consideration of that issue. Instead, we will assume your bona 
fides in asserting that you are the legitimate owner of the subject patent and communicate 

2 

our findings directly with you. To the extent that Mr. Margolin has any interest in this 
matter, formally or informally, we will leave it up to you whether or not to communicate 
with him. 

In light of the prior claim by Mr. Margolin, we consider your license proffer as an 
administrative claim of patent infringement. We turn now to the substance of your claim. In 
response to your initial letter dated July 14, 2008, Mr. McNutt's August 20, 2008 letter 
posed a number of questions, the purpose of which was to enable NASA to fully evaluate 
the details of your claim. Your organization failed to respond to these questions and, further, 
advanced the position that this matter does not involve a new claim (Adams letter to McNutt, 
August 25, 2008). We disagree that this is not a new claim. Nevertheless, NASA proceeds 
— in order to bring closure to this matter — on the basis that this claim centers around 
allegations that infringement arose from activities associated with NASA's X-38 Program, as 
advanced by Mr. Margolin. Accordingly, our investigation of this claim necessarily reflects 
the answers previously furnished by Mr. Margolin in response to NASA's June 11, 2003 
letter to him containing substantially the same set of questions. 

U.S. Patent No. 5,904,724 issued with twenty claims, claims 1 and 13 being the sole 
independent claims. 

In order for an accused device to be found infringing, each and every limitation of the claim 
must be met by the accused device. To support a finding of literal infringement, each 
limitation of the claim must be met by the accused device exactly, any deviation from the 
claim precluding a finding of infringement. See Lantech, Inc. v. Kelp Mach. Co., 32 F.3d 
542 (Fed. Cir. 1994). If an express claim limitation is absent from an accused product, there 
can be no literal infringement as a matter of law. See Wolverine World Wide, Inc. v. Nike, 
Inc., 38 F.3d 1 192, 1 199 (Fed. Cir. 1994). 

In applying these legal precepts, reproduced below are the relevant portions of claims 1 and 
13. 
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Claim 1. A system comprising: 

* # # 

a computer 

### 

said computer is,., for determining a delay time for communicating said flight data between 
said computer and said remotely piloted aircraft, and wherein said computer adjusts the 
sensitivity of said set of one or more remote flight controls based on said delay time, 
(emphasis added.) 

Claim 13. A station for flying a remotely piloted aircraft that is real or simulated 
comprising: 

*** 
a computer 

3 

said computer... to determine a delay time for communicating. . flight control information 
between said computer and [a] remotely piloted aircraft, and said computer to adjust the 
sensitivity of [al set of remote flight controls based on said delay time. ... (emphasis added.) 

NASA has investigated activities surrounding the X-38 program at its Centers that 
conducted X-38 development efforts and has determined that no infringement has occurred. 
This result is compelled because none of NASA's X-38 implementations utilized a computer 
which is "for determining a delay time for communicating said flight data between said 
computer and said remotely piloted aircraft," as required by claim 1 , nor a "computer ... to 
determine a delay time for communicating ... flight control information between said 
computer and [a] remotely piloted aircraft," as required by the limitations of claim 13. 

Given that a computer which measures delay time is lacking from the NASA X-38 
configuration, it follows that the NASA X-38 configuration had no "adjusting of the 
sensitivity of [a] set of one or more remote flight controls based on said delay time", as 
required in claim 1. Similarly, because the NASA X-38 configuration had no "computer to 
determine a delay time for communicating ... flight control information between said 
computer and [a] remotely piloted aircraft, the configuration also had no adjusting of "the 
sensitivity of [a] set of remote flight controls based on said delay time", as called for by 
claim 13. 

For at least the above-explained exemplary reasons, claims 1 and 13 have not been 
infringed. It is axiomatic that none of the dependent claims may be found infringed unless 
the claims from which they depend have been found to be infringed. Wahpeton Canvas Co. 
v. Frontier, Inc., 870 F.2d 1546 (Fed. Cir. 1989). One who does not infringe an independent 
claim cannot infringe a claim dependent on, and thus containing all the limitations of, that 
claim. Id. Thus, none of claims 2-12 and 14-20 have been infringed. 
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NASA's X-38 development efforts ended in 2002. There may also be other features in 
NASA's X-38 development efforts that, upon further analysis, would reveal yet more recited 
claim limitations that are lacking in the NASA configuration related to those efforts. 

We also note as a point of particular significance that the limitations included in claims 1 
and 13 discussed above were added by amendment during the prosecution of the patent 
application. It is clear from an analysis of the patent application file wrapper history that the 
individual prosecuting the application stressed the importance of "the measurement of a 
communication delay in order to adjust the sensitivity of flight controls based on that delay." 
Also noted is the distinguishing arguments that these claims require that there be a 
"computer ... located in the pilot station" and that "at least one real time measurement of the 
delay and some adjustment is contemplated." (See Applicant's Amendment and Remark, 
February 27, 1998 and Response Under 37 C.F.R. § 1.116, July 6, 1998). Clearly, the Patent 
Office Examiner allowed the application based on these prosecutorial arguments. 

We have completed our investigation regarding the claim of patent infringement of U.S. 
Patent No. 5,904,724 and have determined that there is no patent infringement by, or 



unauthorized use on behalf of, NASA. The above detailed discussion explains the basis for 
NASA's analysis and decision regarding the subject administrative claim. 

As an aside, during NASA's investigation, numerous pieces of evidence were uncovered 
which would constitute anticipatory prior knowledge and prior art that was never considered 
by the U.S. Patent and Trademark Office during the prosecution of the application which 
matured into Patent No. 5,904,724. In view of the clear finding of lack of infringement of 
this patent, above, NASA has chosen to refrain from a discussion that would demonstrate, in 
addition to non-infringement, supra, invalidity of the subject patent. However, NASA 
reserves the right to introduce such evidence of invalidity in an appropriate venue, should 
the same become necessary. 

This is a FINAL agency action and constitutes a DENIAL of the subject administrative 
claim for patent infringement. 

Pursuant to 35 U.S.C. § 286, the statute of limitations for the filing of an action of patent 
infringement in the United States Court of Federal Claims is no longer tolled. Thus, any 
further appeal of this decision must be made by filing a claim for patent infringement in the 
United States Court of Federal Claims, pursuant to 28 U.S.C. § 1498(a). 



Sincerely, 
Gary G. Borda 

Agency Counsel for Intellectual Property 
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The Borda letter is not just a material document, it's a smoking gun. 

1. Despite the documents supplied by OTG, and Margolin's confirmation in a telephone 
conversation with Jan McNutt (Office of the General Counsel), that OTG owns the 
subject patent, NASA continues to cast doubt on the legal ownership of the patent. 

We previously noted in a letter dated August 20, 2008 from Mr. Jan McNutt of our 
office addressed to you that NASA believes there are certain irregularities 
surrounding this and collateral assignment documents associated with the subject 
patent. 

2. NASA asserted it had found prior art to invalidate the patent. 

As an aside, during NASA's investigation, numerous pieces of evidence were 
uncovered which would constitute anticipatory prior knowledge and prior art that 
was never considered by the U.S. Patent and Trademark Office during the 
prosecution of the application which matured into Patent No. 5,904,724. In view of 
the clear finding of lack of infringement of this patent, above, NASA has chosen to 
refrain from a discussion that would demonstrate, in addition to non-infringement, 
supra, invalidity of the subject patent. However, NASA reserves the right to 
introduce such evidence of invalidity in an appropriate venue, should the same 
become necessary. 

In order to make this statement, NASA must have produced a patent report showing how 
each reference is directed to the claims in the '724 patent. This patent report is not 
exempt under 5 U.S.C.§552(b)(5) because it is not "inter-agency or intra-agency 
memorandums or letters which would not be available by law to a party other than an 
agency in litigation with the agency;". 

The reason it is not exempt is because "NASA reserves the right to introduce such 
evidence of invalidity in an appropriate venue, should the same become necessary." 

Circulating the patent report solely within NASA or among other federal agencies is not 
an appropriate venue for NASA to use to have a patent declared invalid. The only 
appropriate venues for NASA to challenge the validity of a U.S. Patent are in the U.S. 
Court of Federal Claims and the Court of Appeals for the Federal Circuit. A Court will 
not accept NASA's word that a patent is invalid due to prior art; NASA would be 
required to produce the evidence. 

Since this patent report is material under Margolin's FOIA Request and is not exempt 
under 5 U.S.C.§552(b)(5) Margolin requests NASA immediately hand it over to him. 
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There is another reason NASA needs to hand over the patent report. Although Margolin 
no longer owns the '724 patent he is still the named inventor. By asserting it has evidence 
to invalidate the patent, and then withholding that evidence, NASA has defamed 
Margolin's reputation as an inventor. It also smacks of 1950s McCarthyism (making 
damaging accusations without providing proper evidence). 

Margolin takes such attacks seriously. There is an article in the December 2008 issue of 
AUVSI's Unmanned Systems Magazine entitled Synthetic Vision Technology for 
Unmanned Systems: Looking Back and Looking Forward by Jeff Fox, Michael 
Abernathy, Mark Draper and Gloria Calhoun [Appendix NB58]. 

The article consists of a spurious history of synthetic vision. Many of the listed sources 
are from NASA, such as the HiMat project. [Appendix NB8] (While HiMat produced 
valuable results, it did not use synthetic vision.) 

Margolin responded with the article Synthetic Vision - The Real Story. [Appendix 
NBll 

Although the editor of AUVSI Magazine had promised Margolin the opportunity to 
respond in the magazine, he later refused to even mention the controversy about the 
Abernathy article. [Appendix NB60] 

NASA should be familiar with the name Mike Abernathy (Rapid Imaging Software). He 
provided the synthetic vision system for the X-38 project. 

NASA should also be interested in the statements made on Abernathy' s behalf in a letter 
from Abernathy's law firm to Optima Technology Group dated October 13, 2006. 
[Appendix NAM 3] 

As you know, RIS creates computer software, and does not use or manufacture UAV 
systems or ground control stations. RIS software is used in UAVs to provide situation 
awareness for sensor operators. It is not used for piloting air vehicles. The sensor operator 
does not pilot the aircraft, and instead sits at a separate workstation operating a payload 
containing one or more cameras, which may be controlled using a joystick to point the 
camera package during search or tracking operations. 

As you know, RIS refuses to allow its products to be used as a pilot aid, and RIS product 
licenses specifically prohibit use for piloting. None of RIS's customers use its software for 
piloting, for very good reason . Serious military regulations control placement of anything 
-synthetic vision included- on a pilot workstation. Before anything can be placed on the 
display in front of a pilot, it has to have met stringent criteria (MIL-STD 1787C, DO-178B, 
etc.), it must have been thoroughly ground tested, and it must have been fully flight tested. 
RIS software has never been through this process, and thus is prohibited from use for 
piloting. Accordingly, UAV manufacturers have purchased RIS products for use on the 
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sensor operator console, but none for the pilot console. This is a matter of Army doctrine 
and applies to Shadow, Warrior and Hunter. 

Nor does RIS have its software in a form that would make it marketable for piloting. RIS 
software products are all based on the Microsoft Windows operating system. This offers 
many advantages, but is inappropriate to piloting aircraft because it is a not a POSIX 
compliant real-time operating system. POSIX compliance is required by flight safety 
regulations. To create such a version would entail a one- to two-year conversion program in 
which RIS has not invested. 

It is important to realize that the market for RIS products is quite different from the relaxed 
civilian world. If a military pilot chose to use synthetic vision in spite of military regulations 
or in defiance of a software license agreement, his career would be damaged or destroyed. 
Military pilots cherish their wings and would not consider risking them on something like 
synthetic version. 

Finally, it appears from your correspondence that you regard research activities like NASA's 
X-38 prototypes (before the program was cancelled in 2002) as infringing the Margolin 
patents. This was not the case because of the claim limitations of the Margolin patents. 
However all RIS work for government agencies, including NASA, was authorized and 
consented to by the U.S. Government, and is protected under 28 U.S.C. § 1498(a). As you 
are aware, any remedies you may have are against the government and are circumscribed by 
that statute and related law. 

Although we need not discuss the invalidity of the Margolin patents given the above 
circumstances, you should be aware that both patents were anticipated by profound prior art 
dating back to 1977. If it should ever become necessary, we are confident that both would be 
held invalid. 

(emphasis added) 



He is asserting that Abernathy's synthetic vision software may not be used for piloting an 
aircraft, either remotely or with the pilot onboard. And yet, it was used for remotely 
piloting the X-38. [Appendix NB20] 

From Appendix NB21: 

On December 13th, 2001, Astronaut Ken Ham successfully flew the X-38 from a 
remote cockpit using LandForm VisualFlight as his primary situation awareness 
display in a flight test at Edwards Air Force Base, California. This simulates 
conditions of a real flight for the windowless spacecraft, which will eventually 
become NASA's Crew Return Vehicle for the ISS. We believe that this is the first 
test of a hybrid synthetic vision system which combines nose camera video with a 
LandForm synthetic vision display. Described by astronauts as "the best seat in the 
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house", the system will ultimately make space travel safer by providing situation 
awareness during the landing phase of flight. 

Did NASA really trust the safety of an 

expensive test vehicle (X-38) to a synthetic '; V- , 



vision system using Microsoft Windows? 




To end this section, note that in 5 U.S.C.§552(f): 
(f) For purposes of this section, the term — 

(1) "agency" as defined in section 551 (1) of this title includes any executive 
department, military department, Government corporation, Government 
controlled corporation, or other establishment in the executive branch of the 
Government (including the Executive Office of the President), or any 
independent regulatory agency; and 

(2) "record" and any other term used in this section in reference to information 
includes — 

(A) any information that would be an agency record subject to the 
requirements of this section when maintained by an agency in any format, 
including an electronic format; and 

(B) any information described under subparagraph (A) that is maintained for 
an agency by an entity under Government contract, for the purposes of 
records management. 

Under this definition, neither Margolin nor Optima Technology Group (the owner of 
Claim 1-222) is an "agency." It also means that NASA is required to provide the records 
between NASA and Rapid Imaging Software (Mike Abernathy) which provided the 
synthetic vision system for the X-38 project which was referred to in the Borda letter. 
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3. The basis for NASA's rejection of Claim 1-222 in the Borda letter is that the X-38 
project did not implement one of the elements in the patent claims. 

said computer is,., for determining a delay time for communicating said flight data 
between said computer and said remotely piloted aircraft, and wherein said computer 
adjusts the sensitivity of said set of one or more remote flight controls based on said 
delay time, (emphasis added.) 



To be precise, said computer does more than determine and compensate for time delays. 
Claim 1 says: 

1. A system comprising: 

a remotely piloted aircraft including, 

a position determining system to locate said remotely piloted aircraft's position in 
three dimensions; and 

an orientation determining system for determining said remotely piloted aircraft's 
orientation in three dimensional space; 

a communications system for communicating flight data between a computer and 
said remotely piloted aircraft, said flight data including said remotely piloted 
aircraft's position and orientation, said flight data also including flight control 
information for controlling said remotely piloted aircraft; 

a digital database comprising terrain data; 

said computer to access said terrain data according to said remotely piloted aircraft's 
position and to transform said terrain data to provide three dimensional projected 
image data according to said remotely piloted aircraft's orientation; 

a display for displaying said three dimensional projected image data; and 

a set of one or more remote flight controls coupled to said computer for inputting 
said flight control information, wherein said computer is also for determining a delay 
time for communicating said flight data between said computer and said remotely 
piloted aircraft, and wherein said computer adjusts the sensitivity of said set of one or 
more remote flight controls based on said delay time. 
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Claim 13 says: 

13. A station for flying a remotely piloted aircraft that is real or simulated 
comprising: 

a database comprising terrain data; 

a set of remote flight controls for inputting flight control information; 

a computer having a communications unit configured to receive status information 
identifying said remotely piloted aircraft's position and orientation in three 
dimensional space, said computer configured to access said terrain data according to 
said status information and configured to transform said terrain data to provide three 
dimensional projected image data representing said remotely piloted aircraft's 
environment, said computer coupled to said set of remote flight controls and said 
communications unit for transmitting said flight control information to control said 
remotely piloted aircraft, said computer also to determine a delay time for 
communicating said flight control information between said computer and said 
remotely piloted aircraft, and said computer to adjust the sensitivity of said set of 
remote flight controls based on said delay time; and 

a display configured to display said three dimensional projected image data. 



Is Borda saying that NASA did not determine and compensate for time delays in the X- 
38 synthetic vision flight control loop or simply that NASA did not use a computer to do 
so? If they did not use a computer, what did they use? 

NASA is well aware of the problems caused by failing to compensate for time delays in 
flight control systems. 

When a UAV is manually flown by a remote pilot, failure to compensate for delays in the 
communications link will lead to Pilot-Induced-Oscillation, which frequently leads to the 
loss of the aircraft. 

This is a potential problem in Flight Control Systems even in aircraft with the pilot 
onboard. 
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The article Fly-By- Wire - A Primer for Aviation Accident Investigators (Air Line 
Pilot, February 2000, page 18 By F/O Steve Stowe (Delta), Local Air Safety Chairman, 
Delta Council 16) gives a basic explanation of the Control Systems Engineering analysis 
of the problem. From Appendix NA87: 

Now for the bad news. While FBW technology could make an aerodynamic ally 
unstable aircraft flyable, it can also destabilize an otherwise stable airframe. 

FBW flight control laws may not be stable for all values of gain or phase angle (the 
difference between pilot input and airplane response in terms of frequency; exactly 
opposite would be a 180-degree phase angle) that can be applied. Now costarring 
with static margin as stability factors are "gain margin" and "phase margin"- 
measures of how much additional gain or phase-angle lag are available until the 
system becomes unstable. Computer simulation or flight testing can determine these 
two margins. But these data are often the manufacturer's proprietary information, so 
don't look for it on your weight-and-balance sheet. 

Highly augmented aircraft, in which fly-by-wire transforms the basic aircraft 
aerodynamics, can exhibit cliff-like handling qualities. 

"One reason is that fly-by-wire systems are susceptible to time delay, from a number 
of causes, which can seriously degrade the pilot's ability to control the aircraft. Time 
delay may vary for different sizes or frequencies of inputs. U.S. military standards 
suggest that time delays should be less than one tenth of a second for good handling 
qualities and that loss of control may occur with delays more than one quarter of a 
second (MIL STD 1797)." 

(emphasis added) 

Fly-By- Wire" means the aircraft surfaces are controlled through a computer instead of 
being controlled directly by the pilot. 

From the same article [Appendix NA92]: 

* Time delay-Delay from pilot input to FBW aircraft response. Caused by many 
factors including the effect of filters, computer processing time, task time-sharing by 
computers and signal processors, "higher order" effects of the feedback control 
system, digital sampling effects, and/or actuator rate limiting. Time delays of more 
than 0.25 second can cause enough lag to make the FBW aircraft unstable during 
certain tasks, especially in "high gain" situations. 

(emphasis added) 
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There was a problem with Pilot-Induced-Oscillation during the development of the Space 
Shuttle. The following is from NASA Technical Memorandum NASA-TM-81366 
ANALYSIS OF A LONGITUDINAL PILOT-INDUCED OSCILLATION 
EXPERIENCED ON THE APPROACH AND LANDING TEST OF THE SPACE 
SHUTTLE , Author: J. W. Smith, December 1981. 

From the Introduction (Appendix NA96): 

During the final free flight (FF-5) of the shuttle's approach and landing test (ALT) 
phase, the vehicle underwent pilot-induced oscillations (PIO's) near touchdown (refs. 
1 to 3). The oscillations were present in both the pitch and roll axes and were 
initiated when the pilot made pitch controller inputs in an effort to control sink rate 
by changing pitch attitude. Because the control inputs were large and fairly rapid, the 
elevons rate limited in the pitch axis at the maximum priority rate limit set in the 
computers. The elevon rate limit also limits the vehicle's roll control capability, and 
this was partially responsible for the lateral control problem. 

Several unpublished studies indicate that time delays as well as priority rate limiting 
were a significant factor in the PIO's. A simulator study of the effect of time delays 
on shuttle PIO's is reported in reference 4. 

This report describes the combined effect of pilot input rate limiting and time delays. 
Frequency responses are predicted for various parameters under rate saturated 
conditions by using nonlinear analysis. 

(emphasis added) 

Note that the above references were for Flight Control Systems for aircraft with the pilot 
onboard. When an aircraft is flown manually through a communications link, the delays 
caused by the communications link become part of the flight control system. 

From U.S. Patent 5,904,724 column 8, lines 14 - 36 [Appendix NA142]: 

Flying an RPV is further complicated because there are additional time delays in the 
loop. The computer in the remote aircraft must first determine the aircraft's position 
and orientation. The additional processing for transmitting a secure signal by 
encryption and/or spread spectrum techniques may create additional delays. 
Transmission delay of signals between the remote aircraft and remote pilot station is 
negligible for a direct path. However, if the signals are relayed through other 
facilities the delay time may be appreciable, especially if an orbiting satellite is used. 
There are additional delays in the remote pilot station as the remote aircraft's position 
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and orientation are used to transform the data from the digital database to present the 
pilot with the synthesized 3D projected view from the remote aircraft. In one 
embodiment, the RPV system measures the various delays and modifies the control 
laws used by the computer in the remote pilot aircraft and in the feedback provided 
by the computer in the remote pilot station to the remote pilot. For example, the 
computer may adjust the sensitivity of the User Flight Controls 408 according to the 
delay (e.g., as the delay increases, the computer will decrease the sensitivity of the 
flight controls). The system also displays the measured delay to the remote pilot. 

The issue of time delay in a UAV communications link was addressed in the literature by 
the Master's Thesis Improving UAV Handling Qualities Using Time Delay 
Compensation by Andrew J. Thurling (17 Sep 97-24 Feb 00, AIR FORCE INST OF 
TECH WRIGHT-PATTERSONAFB OH). From Appendix NA1 39: 

Abstract 

This study investigated control loop time delay and its effect on UAV handling 
qualities. Compensation techniques to improve handling qualities in the presence of 
varying amounts of time delay were developed and analyzed. One technique was 
selected and successfully flight-tested on a UAV. 

Flight-testing occurred at a constant flight condition with varying levels of additional 
time delay introduced into the control loop. Research pilots performed a pitch 
tracking task and gave Cooper-Harper ratings and comments. Tracking errors were 
used as a quantitative measure of Pilot/Display/UAV system performance. 

Predictive pitch compensation was found to significantly reduce pilot workload and 
improve Cooper-Harper ratings. Using the predictive display doubled the amount of 
system time delay that research pilots could tolerate while tracking the task bars. 
Overall system tracking performance, however, was not improved. 

Parameter variations of +/- 20% in the aerodynamic model used to generate the 
predictive display produced statistically significant, although not operationally 
significant, changes in both pilot opinion and performance. 

Analysis of flight test data and follow-on simulations resulted in predictor 
improvements that increased predictor accuracy to the point of restoring system 
tracking performance to equal that of the system with no additional time delay. 
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From Appendix NA140: 

Preface 

The effects of control system time delays on manned aircraft handling qualities are 
well understood. Unmanned aircraft have similar control, system delay, but have an 
additional latency caused by the datalink of the human operator's commands from 
control station to aircraft. The purpose of this thesis is to investigate the effects of 
time delay on the handling qualities of Unmanned Aerial Vehicles f UAV) and 
develop compensation strategies to mitigate the adverse effects of the delay. It is my 
hope that with techniques developed and investigated in this thesis future UAV 
operators will be able to employ UAVs from anywhere in the world thus increasing 
the flexibility of this already versatile platform. 

(emphasis added) 



And from the same report (Appendix NA141): 

2.3.4 Time Delay Effects on Handling Qualities. Control difficulties during the 

1977 Space Shuttle Approach and Landing Tests and YF-17 development resulted in 
efforts to investigate whether time delays associated with digital flight computers 
might be a contributing factor to the handling qualities problems. As discussed 
above, delays in flight control systems may come from a variety of sources. The 
effects of phase lag due to higher order effects, or analog time delay, had been 
studied (15) and were relatively well understood. A detailed study of the effects of 
pure delay, transport delay due to digital systems, had yet to be accomplished. In 

1978 a NASA study employed an F-8 fighter aircraft modified with a digital flight 
control system to accomplish a detailed study of the effects of pure time delays on 
aircraft handling qualities (7, 4, 6). In 1979, Hodgkinson and others (29) conducted a 
study on the USAF/Calspan NT-33 inflight simulator in which they tested how 
mismatches between the higher order system and the LOES affected pilot opinion. 
They also investigated how well the delay term, e" ST , in the LOES approximated the 
higher order phase lags and if the difference caused variations in pilot opinion. Both 
studies showed a strong correlation between pilot rating and the magnitude of the 
time delay, see Figures 2.8 and 2.10. The NT-33 data also showed that the 
degradation in pilot rating was similar for both digital transport delay and analog 
delay, or delay due to phase lag from higher order effects. The insidious nature of 
time delay's effects on handling qualities is demonstrated in a pilot comment during 
the F-8 research (7) 

Pilots desire some response immediately upon stick input. It doesn't have to be 
much, but if he doesn't get response, his gains skyrocket. 
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The pilots in the NT-33 study also voiced similar concerns with delay after control 
inputs and the rapidity of the response following the delay. The authors of the F8 
study (7) make a further observation that aircraft dynamics have an impact on system 
sensitivity to time delay. 

(emphasis added) 



So, is Borda saying that NASA did not determine and compensate for time delays in the 
X-38 synthetic vision flight control loop or simply that NASA did not use a computer to 
do so? 

Which is it, because when a UAV is manually flown by a remote pilot, failure to 
compensate for delays in the communications link will lead to Pilot-Induced-Oscillation, 
which frequently leads to the loss of the aircraft. 

Did NASA risk the X-38 by failing to provide compensation for the time delays in the 
synthetic vision flight control loop? 
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Conclusion 

In its very tardy response to FOIA Request 08-270 by Jed Margolin ("Margolin") NASA 
withheld documents, citing 5 U.S.C.§552(b)(5). 

One of the documents that NASA withheld from Margolin is a letter dated March 19, 
2009 that was sent by Gary G. Borda ("Borda") NASA Agency Counsel for Intellectual 
Property to Optima Technology Group ("OTG"). (This document was given to Margolin 
by OTG.) In this letter Borda denies Claim 1-222 regarding NASA's infringement of U.S. 
Patent 5,904,724 ('724) in the X-38 project. 

Margolin's FOIA 08-270 request to NASA was to produce documents relating to Claim 
1-222 and NASA withheld the most material document so far. 

The Borda letter asserts: 

"... numerous pieces of evidence were uncovered which would constitute 
anticipatory prior knowledge and prior art that was never considered by the U.S. 
Patent and Trademark Office during the prosecution of the application which 
matured into Patent No. 5,904,724." 

And states, "... NASA reserves the right to introduce such evidence of invalidity in an 
appropriate venue, should the same become necessary." 

Circulating the patent report solely within NASA or among other federal agencies is not 
an appropriate venue for NASA to use to have a patent declared invalid. The only 
appropriate venues for NASA to challenge the validity of a U.S. Patent are in the U.S. 
Court of Federal Claims and the Court of Appeals for the Federal Circuit. A Court will 
not accept NASA's word that a patent is invalid due to prior art; NASA would be 
required to produce the evidence. 

Therefore, the exemption under 5 U.S.C.§552(b)(5) does not apply. 

Margolin requests NASA produce the evidence that Borda refers to when he asserted: 

"... numerous pieces of evidence were uncovered which would constitute 
anticipatory prior knowledge and prior art that was never considered by the U.S. 
Patent and Trademark Office during the prosecution of the application which 
matured into Patent No. 5,904,724." 

Margolin also requests that NASA show how such materials and/or documents are 
directed to the '724 claims. 
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And, finally, under 5 U.S.C.§552(f) NASA is required to provide the records between 
NASA and Rapid Imaging Software (Mike Abernathy) which provided the synthetic 
vision system for the X-38 project which was referred to in the Borda letter. 



Respectfully, 



Dated: June 10, 2009 

/Jed Margolin/ 

Jed Margolin 
1981 Empire Rd. 
Reno,NV 89521-7430 
775-847-7845 
jm@imargolin.com 
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Sf UNt TED STATES 
POSTAL SERVICE, 



Date: 06/12/2009 



Jed Margolin: 



The following is in response to your 06/1 1/2009 request for delivery information on your 
Express Mail(R) item number EQ98 521 1 585U S. The delivery record shows that this item 
was delivered on 06/12/2009 at 07:08 AM in WASHINGTON, DC 20546 to T JACKSON. The 
scanned image of the recipient information is provided below. 



Signature of Recipient: 



ature 
< 


"1 






tied 
ne 










very 
loess 





Thank you for selecting the Postal Service for your mailing needs. If you require additional 
assistance, please contact your local Post Office or postal representative. 



Sincerely, 

United States Postal Service 
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Jed Margolin 1981 Empire Rd. Reno, NV 89521-7430 

Phone: 775-847-7845 Email: im@imartiolin.com July 21, 2009 



Mr. Randolph Harris 

NASA Office of the General Counsel 

300 E St. SW 

Washington, DC 20546 

Phone: (202) 358-2450 

Fax: (202) 358-2741 

Email: randolph.harris-l@nasa.gov 

Dear Mr. Harris. 

As per our conversation today please confirm that NASA refuses to waive legal service unless the Complaint 
and Summons is sent to NASA by Certified USPS mail, and will not waive legal service if it is sent by USPS 
Express Mail. 

As I explained during our conversation, I sent Acting Administrator Scolese a certified letter in April which 
USPS did not deliver, and which USPS could not find. Their explanation was that Certified Mail is only 
scanned into their tracking system when it is mailed and when it is delivered. If it is lost in transit it cannot 
be tracked. 

In addition, according to USPS, Certified Mail is sent to New Jersey to be irradiated (delaying delivery and 
increasing the chances of being lost) while Express Mail is not. 

As a result I do not consider Certified USPS mail to be a viable means of sending a Complaint and Summons 
to NASA. 

If NASA refuses to waive service by Express Mail my only option will be to use a Process Server. 

BTW, according to Rule 4 of the Federal Rules of Civil Procedure (December 1, 2008) it looks like I also 
have to serve: 

1 . The Attorney General of the United States, Washington, DC; and 

2. The United States attorney for the district for the district where the action is brought. That would be 
The United States District Court, District of Nevada-Reno. 

If this is correct, please give me the name and address for the United States attorney for the District of 
Nevada-Reno. 

Sincerely yours, 
/Jed Margolin/ 
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Jed Margoli n 

From: "Mcnutt, Jan (HQ-MCO0O)" <jan.mcnutt@nasa.gov> 

To: "Jed Margolin" <jm@jmargolin.com> 

Sent: Friday, July 24, 2009 1 0:42 AM 

Subject: FOIA Appeal of FOIA No. 2008-270 

Dear Mr Margolin, 

I've been informed that you are in contact with Mr. Randolph Harris of our office concerning the subject FOIA 
appeal. I have been assigned to respond to your appeal and as it stands, we are now past due in our response to 
yau. I apologize for the delay and am officially requesting an extension for NASA to respond to your FOIA 
appeal. I would like to ask for a 20 day extension from the action due date that I received, which was July 17, 
2009, which would require us to provide you with a response by August 6, 2009. We have every intention of 
providing you with a proper response, but circumstances have been such that we have not been able to process 
the response in the allotted time 

Thank you for your consideration of an extension in this matter. 

Regards, 

Jan S. McNutt 

Senior Attorney (Commercial) 
Office of the General Counsel 
NASA Headquarters 
Suite 971 1 
300 E Street, SW 
Washington, DC 20546- 0001 
(202) 358-0632 
Jan.McNutt@nasa.gov 
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Jed Margolin 



From: "Jed Margolin" <jm@jmargolin.com> 

To: "Mcnutt, Jan (HQ-MC0O0)" <jan.mcnutt@nasa.gov> 

Sent: Friday, July 24, 2009 1 2:53 PM 

Subject: Re: FOIA Appeal of FOIA No. 2008-270 

Mr. McNutt, 



You wrote: 

> I've been informed that you are in contact with Mr. Randolph Harris of our office concerning the 
subject FOIA appeal. I have been assigned to respond to your appeal and as it stands, we are now past 
due in our response to you. I apologize for the delay and am officially requesting an extension for NASA 
to respond to your FOIA appeal. I would like to ask for a 20 day extension from the action due date that 
I received, which was July 17, 2009, which would require us to provide you with a response by August 6, 
2009. We have every intention of providing you with a proper response, hut circumstances have been 
such that we have not been able to process the response in the allotted time. 



My response: 

1 . NASA failed to respond or ask for an extension within the 20 day statutory period. 

2. NASA has been acting in bad faith toward me for the past six years and some months. 

3. You have personally acted in bad faith toward me by taking improper advantage of (and my regretting) every 
courtesy I have ever extended to you. 

4. When I asked Mr. Harris if NASA would accept Legal Service by Express Mail, he said, "No." Only by 
Certified Mail. I explained that when I sent NASA Certified Mail in April, the USPS failed to deliver it and was 
unable to determine how it was lost or where. As a result, I do not consider Certified Mail reliable and I will 
have to pay a process server to serve Administrator Bolden. Mr. Harris still said, "No." 

5. Mr. Harris has failed to respond to my email (and later fax) asking him to confirm what he told me in our 
telephone conversation (that NASA will not accept Legal Service by Express Mail). 

6. Mr. Harris said NASA's response to my FOIA Appeal will be to send me a bunch of documents, but he 
didn't know when. He guessed "7 days." Your characterization of NASA's response contains no promise of 
documents, only "a proper response." 



And so, Mr. McNutt, my answer to you is "No." 



Sincerely yours, 



Jed Margolin 
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— Original Message 

From: Mcnutt, Jan (HQ-MC00O) 
To: Jed Margolin 

Sent: Friday, July 24, 2009 10:42 AM 
Subject: FOIA Appeal of FOIA No. 2008-270 

Dear Mr. Margolin, 

I've been informed that you are in contact with Mr. Randolph Harris of our office concerning the subject FOIA 
appeal. I have been assigned to respond to your appeal and as it stands, we are now past due in our response 
to you. I apologize for the delay and am officially requesting an extension for NASA to respond to your FOIA 
appeal. I would like to ask for a 20 day extension from the action due date that I received, which was July 1 /, 
2009, which would require us to provide you with a response by August 6, 2009. We have every intention of 
providing you with a proper response, but circumstances have been such that we have not been able to process 
the response in the allotted time 

Thank you for your consideration of an extension in this matter. 

Regards, 

Jan S. McNutt 

Senior Attorney (Commercial) 
Office of the Genera! Counsel 
NASA Headquarters 
Suite 9T1 1 
300 E Street, SW 
Washington, DC 20546-0001 
(202) 358-0632 
Jan.McNutt@nasa.gov 
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Synthetic Vision Technology 
for Unmanned Aerial Systems: 

Looking Back and Looking Forward 

By Jeff Fox, Michael Abernathy, Mark Draper and Gloria Calhoun 



Using computers and terrain databases to generate a 
simulated, real-time, three-dimensional view of an 
environment — otherwise known as synthetic vision — has 
been applied to unmanned aircraft systems for three decades. 

More recently it has evolved away from being a piloting aid to a 
potentially powerful rool for sensor operators, 'lechnology observers 
expect it can help offset many factors that currently compromise the 
usefulness of UAS video imagery: narrow camera field of view, 
degraded datalinks, poor environmental conditions, limited 
bandwidth and highly cluttered visual scenes such as those found in 
urban areas. 

With synthetic vision technology, information can be pulled from 
databases (of terrain elevation, cultural features, maps, photo 
imagery) and combined with data from networked sources, all of 
which can be represented as com purer- generated imagery and 
symbology and overlaid on a dynamic video image display. The 
imagery and symbology appears to coexist with real objects in the 
scene, allowing an operator to cut through the clutter and maintain 
situational awareness of the environment. 

There is a large body of research from the 1 970s to the present that 
addresses the application of synthetic vision to manned and 
unmanned aircraft. In the interest of brevity, this article will focus 
on select systems that were important enablers toward UAS 
synthetic vision systems. 

The story begins in the 1970s when the use of computers to create 
3D real-time, out-the-window synthetic environments was 
beginning to see wide acceptance for training pilots of manned 
aircraft, Computer graphics company Evans and Sutherland 
(E&S), of Salt Lake City, Utah, had seen the commercial potential 
for flight simulation and had introduced special-purpose graphics 
computers, like their Picture System, which transformed and 
projected 3D terrain data as simple 3D polygons to a pilot's 
perspective view in real-time. In 1975, an engineering student 
named Bruce Artwick wrote "Flight Simulator" for the Apple II 
computer. He formed a company and in 1980 marketed the 
product that ultimately became Microsoft Flight Simulator. 

This emergence of computer flight simulation in the 1 970s appears 
to have sparked a monumental amount of research. The U.S. Air 
Force began its Visually Coupled Airborne Systems Simulator 
(VCASS) program, with a particular eye toward future-generation 
fighter aircraft ("VCASS: An Approach to Visual Simulation," 
Kocian, D„ 1977). NASA was developing synthetic vision for the 
Super Sonic Transport and for its High Maneuverability Aircraft 




NASA's HiMAT remotely piloted vehicle after flight nt Dryden Flight Research Center. Photo courtesy of 
NASA. 



Testbcd {HiMAT} remotely piloted vehicle (RPV) program. 
Educational institutions studied the limitless new possibilities for 
virtual reality human-machine interfaces. By the mid-1980s, 
synthetic vision for RPV simulation was even commercially 
available for radio control aircraft hobbyists. 

In 1977, NASA researcher Charles Knox published "Pathway- 
in-the-Sky Contact Analog Piloting Display," which included a 
complete design for a synthetic vision system. It featured a computer 
that projected a 3D view of the terrain given an aircraft's position 
and orientation. This out-the-window perspective view was 
displayed on a CRT type display. Such displays were called "Pictorial 
Format" avionics systems, but we recognize them as containing all 
of the essential elements of a modern synthetic vision display. 

In 1979, the U.S. Air Force completed its "Airborne Electronic 
Terrain Map Applications Study" and in 1981 published "The 
Electronic Terrain Map: A New Avionics Integrator" describing 
how a computerized terrain database could be displayed as an out- 
the-window 3D view allowing the pilot to "see" even at night and 
in other limited visibility situations. 

Also in 1979, the Air Force published research identifying human 
factors problems that would have to be overcome in RPV cockpit 
design ("Visual-Proprioceptive Cue Conflicts in the Control of 
Remotely Piloted Vehicles" by Reed in 1977). NASA would use 
this in the design of the HiMAT RPV 3D visual system in 1984. 

Pictorial format avionics (i.e., synthetic vision) formed a key 
ingredient of the Air Force Super Cockpit concept. This program 
included a bold future vision in which "the pilot need not 
be present in the actual vehicle which he is piloting since with 
the appropriate data links a 'remote* super cockpit would provide 
the visual and aural 'telepresence' cues as if he were located ui the - 
vehicle," according to Air Force researcher TonvF^rteSs/ 

CONTINUED 0M.tftt'».~ 
{Jmottcd Syitani - DKMbr 7008 27 
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Synthetic Vision ... continued 




HiMAT: RPV with Synthetic Vision 

In 1984, NASA researcher Shahan Sarrafian published research that 
investigated synthetic vision for lateral control during RPV 
landings. These tests featured the HiMAT vehicle, flown at Dryden 
Plight Research Center. These aircraft were dropped from a B-52 
and remotely piloted from a ground station to a landing on the 
lakebed. The vehicle had a nose camera which produced video that 
could be shown in the remote cockpit, allowing the comparison of" 
nose camera imagery versus synthetic vision during pilot testing. 

Vehicle position was computed using radar computations along 
with a radio altimeter. Electro-mechanical gyroscope systems were 
installed onboard the aircraft and measured the three-dimensional 
attitude of the vehicle. The position and attitude were down-linked 
from the aircraft to a remote cockpit, and pilot control inputs were 
up-linked from the remote cockpit via the radio communication 
system. 

The remote cockpit included a 
joystick and rudder controls 
connected to the computer and 
control signals were uplinked 
to the UAV. The computer 
compensated for delays in the 
control/ communications loop. 

The Edwards Air Force Base 

dry lake bed and runway were 

... .. The HiM RPV femole cockpit showing synthetic 

represented in three dimensions ^ disp | oy py mm/ rf mL 

in the terrain database as 

polygons (triangles and rectangles). An E&S Picture System 
computer transformed the terrain in the database into a projected 
3D out-the-window view at the pilot cockpit. Finally, the projected 
3D view was displayed on an E&S Calligraphic video display 
system capable of 4000 lines of resolution. According to the pilots 
participating in the study, the synthetic vision compared well to the 
nose camera view. By the mid 1990s, NASA had migrated the RPV 
synthetic vision concept used on HiMAT to PC computers for the 
X-36 and X-38 flight demonstration vehicles. 

One of the early uses of synthetic vision for UAVs — then most 
often called RPVs — was recreational simulation. In 1986, 
Ambrosia Microcomputer Products of Willowbrook, ILL, 
introduced RC AeroChopper, a radio controlled aircraft simulator 
which enabled pilots to learn to fly a remotely controlled aircraft, 
without risk to their actual vehicle. According to the "AeroChopper 
Owner's Manual" (Stern, 1986), the product accepted aileron, 
elevator, rudder, and throttle pilot inputs via joysticks to control the 
simulated aircraft. The product also contained data files containing 
a 3D terrain database provided with AeroChopper representing the 
earth's surface as well as buildings and obstructions. 

The software was run on a computer (an Amiga for example) and 
was connected to the flight controls and communicated the aircraft 
position and attitude to the user. The computer used the terrain 
data to create a projected view of the aircraft and its environment 
in three dimensions. Like most visual simulations of its time, the 
program used relatively few polygons to represent the terrain and 
man-made objects and so looks crude by today's standards. 



Synthetic Vision for Sensor Operations 

Although most of the historical focus with synthetic vision has been 
on aiding flight management, recenr efforts have focused on how 
synthetic vision can aid UAS sensor operator functions. 

Ongoing research at the U.S. Air Force Research Laboratory's 
Human Effectiveness Directorate is exploring how to improve the 
usefulness of video imagery to UAS sensor operators. The overall 
objective is to determine the value of combining synthetic vision 
imagery/symbology with live camera video presented on a UAS 
control station camera display. 

One research study evaluated the utility of computer-generated 
video overlays for four different task types: controlling the camera 
to locate specific ground landmarks in the 360 degree area 
surrounding the loitering UAV; designating multiple ground 
targets marked with synthetic symbology; tracing a synthetically 
highlighted ground convoy route with the UAV camera boresight; 
and reading text from synthetic overlaid symbology. 

The UAS telemetry update rate was manipulated from 0.5 Hz to 24 
Hi. The results indicated the potential of synthetic symbology 
overlay for enhancing situation awareness, reducing workload and 
improving the designarion of points of interest at nearly all the 
update rates evaluated and for all four task types. However, daut 
across the task types indicated that update rates greater than 2-4 H? 
generally resulted in improved objective performance and a 
subjective sense that the symbology was useful. 

A second research area focused on a picture-in-picture (PIP) 
concept where video imagery is surrounded by a synthetic- 
generated terrain imagery border on the physical camera display, 
increasing the operator's instantaneous field-of-view, Experimental 
data showed that the PIP helps mitigate the "soda-straw effect," 
reducing landmark search time and enhancing operator situation 
awareness. In an evaluation examining the impact of PIP display 
size and symbology overlay registration errors, results indicated that 
performance on a landmark search task was particularly better with 
the more compressed video imagery, reducing average designation 
time by 60 percent. Also, the registration error between the virtual 
flags and their respective physical correlates was less critical with the 
PIP capability enabled. The details were published in "Picture-in- 
Picture Augmentation of UAV Workstation Video Display" by- 
Gloria Calhoun and others in 2007. 

The recent availability of sophisticated UAS autopilots capable of 
autonomous flight control has fundamentally changed the 
paradigm of UAS operation, potentially reducing the usefulness of 
synthetic vision for supporting UAS piloting tasks. At the same 
time, research has demonstrated and quantified a substantial 
improvement in the efficiency of sensor operations through the use 
of synthetic vision sensor fusion technology. We expect this to 
continue to be an important technology for UAS operation. 

Jeff Fox is Flight Operations Engineer at NASA Johnson Space Center. 
Michael Abernathy is Director of Development with Rapid Imaging 
Software, Inc. Mark Draper and Gloria Calhoun are Senior Research 
Scientists at the Air Force Research Laboratory, Wright-Patterson Air 
Force Base, Ohio. 
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Introduction 



This is in response to the article Synthetic Vision Technology for Unmanned Systems: Looking 
Back and Looking Forward by Jeff Fox, Michael Abernathy, Mark Draper and Gloria Calhoun which 
appeared in the December 2008 issue of AUVSI's Unmanned Systems {page 27). (Ref. 1} 

The AUVSI Authors have used the term "synthetic vision" so loosely that many readers will believe it was 
invented long before it actually was. This is an important issue. Aerospace is a field where precision and 
accuracy is critical. There are also patent rights involved. In the interests of full disclosure I am the listed 
inventor on several patents relating to synthetic vision and there is a patent infringement disagreement 
between the owner of the patents (Optima Technology Group) and the company that one of the AUVSI 
Authors is affiliated with (Rapid Imaging Software). 



What Is Synthetic Vision? 

The term "Synthetic Vision" originally meant anything that you put up on a video display. 

For example, there is U.S. Patent 5,593,1 14 Synthetic Vision Automatic Landing System issued 
January 14, 1997 to Ruhl (Assignee McDonnell Douglas Corporation). {Ref. 2\ 

From Column 2, lines 16 - 27: 

The instant invention is an Enhanced or Synthetic Vision (also called Autonomous) Landing 
System (E/SV). This system allows the pilot to view the approach scene with the use of a 
forward looking radar or equivalent sensor which provides the means of identifying the runways 
and the airport and land the aircraft using the automatic landing systems on virtually all types of 
aircraft. A pilot effectively turns the flight task during zero visibility or other low visibility 
weather conditions into a synthetic "see to land" approach because the image from the forward 
looking sensor provides sufficient detail to turn any instrument landing into what appears to be a 
visual landing. 

In this patent Enhanced or Synthetic Vision is a display of the data from a forward looking radar or 
equivalent sensor. 

This was also the FAA's definition at the time, in their Synthetic Vision Technology Demonstration, 
Volume 1 of 4, Executive Summary (Ref 3} . From PDF page 10: 

1.1 BACKGROUND 

In 1988 the Federal Aviation Administration (FAA), in cooperation with industry, the United 
States Air Force (USAF), the Navy, and several other government organizations initiated an 
effort to demonstrate the capabilities of existing technologies to provide an image of the runway 
and surrounding environment for pilots operating aircraft in low visibility conditions. This effort 
was named the Synthetic Vision Technology Demonstration (SVTD) program. Its goal was to 
document and demonstrate aircraft sensor and system performance achieved with pilots using 
millimeter wave (MMW) radar sensors, a forward-looking infrared (FLIR) sensor, and a head-up 
display (HUD). 

And from PDF pages 11,12: 

A70 



Case3:09-cv-00421-LRH-VPC Document 1-1 Filed 07/31/09 Page 70 of 128 



3 

1.2. OBJECTIVE 

The objective of the Synthetic Vision Technology Demonstration program was to develop, 
demonstrate, and document the performance of a low-visibility, visual-imaging aircraft landing 
system. The experimental Synthetic Vision System components included on-board imaging 
sensor systems using millimeter-wave and infrared technology to penetrate fog, and both head- 
up (HUD) and head-down (HDD) displays. The displays presented the processed raster image of 
the forward scene, combined with suitable avionics-based stroke symbology for the pilot's use 
during a manually flown approach and landing. The experimental system, sometimes referred to 
as a functional prototype system, included all the functions (in prototype form only) required to 
accomplish precision, non-precision, and non-instrument approaches and landings in low 
visibility weather conditions. 

In the AUVSI Authors' own article they equate "pictorial format avionics" with "synthetic vision." 
[Paragraph 10]: 

Pictorial format avionics (i.e., synthetic vision) formed a key ingredient of the Air Force Super 
Cockpit concept. 

Boeing's report Multi-Crew Pictorial Format Display Evaluation {Ref. 4} describes what Pictorial 
Format means (PDF Page 17): 

The Multi-Crew Pictorial format Display Evaluation Program is the third in a series of contracted 
efforts, sponsored primarily by the Air Force Flight Dynamics Laboratory, Crew Systems 
Development Branch, (AFWAL/FIGR). In the first of these efforts, conceptual displays were 
developed for six primary fighter crew station functions: primary flight , tactical situation , stores 
management , systems status , engine status , and emergency procedures (Jauer and Quinn, 1982). 

In the second contract, Pictorial Format Display Evaluation (PFDE), the Boeing Military 
Airplane Company continued the development beyond the paper formats of the earlier program 
and implemented the results in a piloted simulation. Two simulation studies were conducted to 
evaluate the usability and acceptability of pictorial format displays for single-seat fighter aircraft; 
to determine whether usability and acceptability were affected by display mode — color or 
monochrome; and to recommend format changes based on the simulations. In the first of the two 
PFDE studies, pictorial formats were implemented and evaluated for flight, tactical situation, 
system status, engine status, stores management, and emergency status displays. The second 
PFDE study concentrated on the depiction of threat data . The number of threats and the amount 
and type of threat information were increased. Both PFDE studies were reported in Way, 
Hornsby, Gilmour, Edwards and Hobbs, 1984. 

Pictorial Format Avionics is pictures. That explains why it is called Pictorial Format Avionics. 
Why can't we use the term "Synthetic Vision" to mean anything we want it to mean? 

1 . It is sloppy. 

2. The FAA has a definition for "Synthetic Vision" and if you want an FAA type certificate for your 
Synthetic Vision product you have to use their definition. 
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IRef. 5 - FAA current definition of synthetic vision} 

Synthetic vision means a computer-generated image of the external scene topography from the 
perspective of the flight deck that is derived from aircraft attitude, high-precision navigation 
solution, and database of terrain , obstacles and relevant cultural features. 

{Emphasis added} 

IRef. 6 - FAA Synthetic Vision is based on a Digital Elevation Database} 
"Everyone gets their data from the same original source." 

"If accuracy of data base must be validated then SV is unapproveable." 

"Current resolution tends to round-up the elevation data so that small errors are not as significant 
and on the conservative side." 

{Emphasis added} 

Therefore, Synthetic Vision means a computer-generated image of the external scene topography from 
the perspective of the flight deck that is derived from aircraft attitude, high-precision navigation solution, 
and digital terrain elevation database, obstacles and relevant cultural features. 

Implicit in this is that in order for the external scene topography to be viewed from the perspective of the 
flight deck it has to be a 3D projected view and that the digital terrain elevation database must represent 
real terrestrial terrain, as opposed to terrain that is simply made up. 



Digital Terrain Elevation Database 

The Digital Terrain Elevation Database is also called the Digital Elevation Database or Digital 
Elevation Model. From Ref. 7 : 

The USGS Digital Elevation Model (DEM) data files are digital representations of cartographic 
information in a raster form. DEMs consist of a sampled array of elevations for a number of 
ground positions at regularly spaced intervals. These digital cartographic/geographic data files 
are produced by the U.S. Geological Survey (USGS) as part of the National Mapping Program 
and are sold in 7.5-minute, 15-minute, 2-arc-second (also known as 30-minute), and 1-degree 
units. The 7.5- and 15-minute DEMs are included in the large scale category while 2-arc-second 
DEMs fall within the intermediate scale category and 1-degree DEMs fall within the small scale 
category - (Source: USGS) 

The Digital Elevation Model was substantially improved by STS-99 when Endeavour's international crew 
of seven spent 1 1 days in orbit during February 2000 mapping the Earth's surface with radar 
instruments. IRef. 8) 
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Displaying the Digital Elevation Database 

Now that we have a Digital Elevation Database consisting of a sampled array of elevations for a number 
of ground positions at regularly spaced intervals, what do we do with it? The database is just elevation 
points. 

If you display only points there is no way to remove "hidden points" because there are no surfaces to test 
them against. (Things can only be hidden behind surfaces.) The result is a jumble which looks like this 
(the only useful features are the highest peaks): 




This following picture shows the same scene rendered in polygons. (The polygons are crude because I 
had only a few colors to work with and there is no clipping, only polygon sorting): 
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After you have used the digital elevation points to produce polygons you can shade and blend the 
polygons so that the underlying polygons may no longer be obvious. Honeywell did an excellent job in 
their IPFD (Instrument Primary Flight Display) {Ref. 9} : 
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NASA HiMAT 

The AUVSI Authors have gone to considerable lengths to persuade readers that NASA's HiMAT project 
was Synthetic Vision [Paragraphs 1 1 - 14]. It wasn't. 

HiMAT - Summary 

Sarrafian (Ref. 11} 

1. "The vehicle was flown with cockpit display instruments until the landing approach phase of the flight 
when the camera aboard the aircraft was activated to provide the pilot with a television display during the 
approach." 

2. During the operational phase of the HiMAT program, a simulator was used to adjust the control laws 
for the primary control system. The display presented to the pilot of this simulated system was a display 
of an instrument landing system (ILS). 

3. Separately , a study was undertaken to compare evaluations of pilots using a simulated visual display 
of the runway scene and a simulated ILS display with the results of actual flight tests, using the HiMAT 
aircraft as a representative remotely piloted research vehicle. 

There is no mention of a terrain database or any suggestion that the simulated visual display of the 
runway scene was ever used to control a real aircraft. It was never anything other than a simulation. 



From Evans and Schilling fRef. 13}: 

Visual Landing Aid 

Actual. - Cues to the pilot during landing included the cockpit instruments, ILS/glideslope error 
indicators, television transmission from the vehicle, calls on the radio from the chase pilot, and space- 
positioning calls from the flight-test engineer. 

Simulation model . - For most of the program, the landing cues for the pilot in a HiMAT simulation 
included only the instruments, mapboards, and the ILS/glideslope error indicators. Although these are 
all valid cues, they could not achieve the same effect as the television transmission used in actual 
flight. During flight, as soon as the pilot can identify the runway, his scan focuses more on the 
television picture and less on the cockpit instruments. To help alleviate this lack of fidelity in the 
simulation, a display of the runways on the dry lakebed was developed on a recently purchased Evans 
and Sutherland Graphics System. 
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HiMAT Details 

From NASA's description of the HiMAT project {Ref. 10} : 
Highly Maneuverable Aircraft Technology 

From mid-1979 to January 1983, two remotely piloted, experimental Highly Maneuverable Aircraft 
Technology (HiMAT) vehicles were used at the NASA Dryden Flight Research Center at Edwards, 
Calif., to develop high-performance fighter technologies that would be applied to later aircraft. 
Each aircraft was approximately half the size of an F-16 and had nearly twice the fighter's turning 
capability. 

and, later: 

The small aircraft were launched from NASA's B-52 carrier plane at an altitude of approximately 
45,000 feet. Each HiMAT plane had a digital on-board computer system and was flown remotely 
by a NASA research pilot from a ground station with the aid of a television camera mounted in the 
cockpit. There was also a TF-104G chase aircraft with backup controls if the remote pilot lost 
ground control. 

NASA's article says it was flown remotely by a pilot using a television camera in the aircraft. It does not 
say it was flown using what is now known as synthetic vision. (As previously explained, the definition of 
the term "synthetic vision" has changed over the years.) 

It does say: 

Dryden engineers and pilots tested the control laws for the system, developed by the contractor, in a 
simulation facility and then in flight, adjusting them to make the system work as intended. 

and that is where the AUVSI Authors have gone astray, whether deliberately or through poor scholarship. 

The AUVSI Authors cite the report by Shahan Sarrafian, "Simulator Evaluation of a Remotely Piloted 
Vehicle Lateral Landing Task Using a Visual Display." There are two Sarrafian reports with that title, 
one dated May 1984; the other dated August 1984. See Ref. 11 which contains links to the reports as 
well as to mirrored copies. The August 1984 report has been converted to text to make it easy to search 
and to quote from. 

The title of the Sarrafian report gives an accurate description of his project, "Simulator Evaluation of a 
Remotely Piloted Vehicle Lateral Landing Task Using a Visual Display." 

it was a simulation. 

Here is the Introduction from the report. It's a little long but it describes the heart of the matter. I have 
underlined the parts that are especially relevant. 

Introduction 

The remotely piloted research vehicle (RPRV) is a tool that can be used for exploring unproven and 
advanced technologies without risking the life of a pilot. The flight testing of RPRVs(l) allows programs 
to be conducted at a low cost, in quick response to demand, or when hazardous testing is required to 
assure the safety of manned vehicles. Yet this type of testing must be performed by the most versatile 
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system available - the pilot. The pilot has the same responsibilities and tasks as if he were onboard the 
aircraft; this includes guiding the vehicle to a safe landing. The only difference is that he must accomplish 
this final task from a ground-based cockpit. 

The highly maneuverable aircraft technology (HiMAT) aircraft (Fig. 1) is a remotely piloted research 
vehicle that has completed flight tests to demonstrate advanced fighter technologies at NASA Ames 
Research Center's Dryden Flight Research Facility. The HiMAT vehicle is a 0.44-scale version of an 
envisioned small, single-seat fighter airplane. The mission profile of HiMAT (Fig. 2) included a launch 
from a B-52 aircraft and the acquisition of flight test data. The vehicle was then flown by a NASA test 
pilot in a fixed ground-based cockpit to a horizontal landing on the Edwards dry lakebed. The vehicle was 
flown with cockpit display instruments until the landing approach phase of the flight when the camera 
aboard the aircraft was activated to provide the pilot with a television display during the approach. 

During the operational phase of the HiMAT program, the lateral-stick gearing gain used in the aircraft 
a pproach was altered from a variable gain schedule (derived from simulation) to a constant gain schedule. 
The schedules were changed in response to pilot complaints about oversensitivity in the lateral stick that 
required high pilot compensation. Before the modified gain schedule was implemented into the primary 
control system (PCS), it was evaluated in the HiMAT simulator using an instrument landing system (ILS) 
display; the schedule was found to be satisfactory. Postflight comments from HiMAT pilots indicated that 
the handling qualities during landing approach were significantly improved as a result of the modified 
gain schedule. 

In a separate development, a visual display that was used for engineering purposes was implemented 
into the simulator during the latter portion of the flight test program when simulation was no longer 
required to support the remaining flights. While the addition of a visual display is known to significantly 
improve the fidelity of a simulation system, the need for such a system in RPRV simulation at Ames 
Dryden was felt to be reduced since pilots had an opportunity to conduct proficiency flights with an 
RPRV Piper Comanche PA-30 aircraft. Nevertheless, when a visual display became available in the 
simulation laboratory, a decision was made to determine the effectiveness of this type of visual display in 
the simulation of visual RPRV flight. The RPRV evaluation described in this paper was designed to focus 
on the utility of a visual display of this type while studying the influence of changes in lateral-stick 
gearing gains of remotely piloted research vehicle handling qualities during simulated approaches and 
landings. This study was undertaken to compare evaluations of pilots using a simulated visual display of 
the runway scene and a simulated ILS display with the results of actual flight tests, using the HiMAT 
aircraft as a representative remotely piloted research vehicle. 

What this says is: 

1. "The vehicle was flown with cockpit display instruments until the landing approach phase of the flight 
when the camera aboard the aircraft was activated to provide the pilot with a television display during the 
approach." 

2. During the operational phase of the HiMAT program, a simulator was used to adjust the control laws 
for the primary control system. The display presented to the pilot of this simulated system was a display 
of an instrument landing system (ILS). 

3. Separately , a study was undertaken to compare evaluations of pilots using a simulated v isual display 
of the runway scene and a simulated ILS display with the results of actual flight tests, using the HiMAT 
aircraft as a representative remotely piloted research vehicle. 
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There is no mention of a terrain database or any suggestion that the simulated visual display of the 
runway scene was ever used to control a real aircraft. It was never anything other than a simulation. 

Sarrafian does not show a picture of the ILS display. He probably assumed that anyone reading the 
report in 1984 would know what one looks like. 

The following is a modern picture and an explanation of an ILS display from NASA {Hef. 12} . Note that 
the sky above the horizon line is blue; the ground below the horizon line is brown. There is no depiction 
of terrain. This looks a great deal like what is now known as a Primary Flight Display. 



Instrument Landing System (ILS) 

An aircraft on an instrument landing approach has a cockpit with computerized instrument 
landing equipment that receives and interprets signals being from strategically placed stations on 
the ground near the runway. This system includes a "Localizer 1 ' beam that uses the VOR 
indicator with only one radial aligned with the runway. The Localizer beam's width is from 3° to 
6°. It also uses a second beam called a "glide slope" beam that gives vertical information to the 
pilot. The glide slope is usually 3° wide with a height of 1 .4°. A horizontal needle on the 
VOR/TLS head indicates the aircraft's vertical position. Three marker beacons (outer, middle and 
inner) are located in front of the landing runway and indicate their distances from the runway 
threshold. The Outer Marker (OM) is 4 to 7 miles from the runway. The Middle Marker (MM) is 
located about 3,000 feet from the landing threshold, and the Inner Marker (IM) is located 
between the middle marker and the runway threshold where the landing aircraft would be 100 
feet above the runway. 

The VOR indicator for an ILS system uses a horizontal needle in addition to the vertical needle. 
When the appropriate ELS frequency is entered into the navigation radio, the horizontal needle 
indicates where the aircraft is in relation to the glide slope. If the needle is above the center mark 
on the dial, the aircraft is below the glide slope. If the needle is below the center mark on the 
dial, the aircraft is above the glide slope. 
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The following is a picture of the image Sarrafian produced in his simulator {Figure 9 - Simulated landing 
approach conditions on glideslope): 




The display was created with an Evans and Sutherland Picture System fHef. 16} using a calligraphic 
monitor. The term calligraphic means that the system only drew lines and dots. This type of system is 
also called Random Scan because the electron beam in the CRT can be moved anywhere on the 
screen, as opposed to a Raster Scan system, which draws a raster. Atari's term for Random Scan was 
XY or Vector and was used in several games in the late 1970s and early 1980s such as Asteroids, 
BattleZone, and Star Wars. 

The solid areas are filled-in by drawing lots of lines. 

The lines above the horizon are presumably meant to indicate the sky. The grid lines are presumably 
meant to indicate the ground. There is no suggestion that the grid lines are produced from a digital 
elevation database. There would be no reason to use a digital elevation database because the system 
was used only to simulate landings. (Indeed, the name of the study is " Simulator Evaluation of a 
Remotely Piloted Vehicle Lateral Landing Task Using a Visual Display.") 
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Another HiMAT report is THE ROLE OF SIMULATION IN THE DEVELOPMENT AND FLIGHT TEST 
OF THE HIMAT VEHICLE by M. B. Evans and L. J. Schilifng {Ret 13} . 

From Evans and Schilling: 

Visual Landing Aid 

Actual. - Cues to the pilot during landing included the cockpit instruments, ILS/glideslope error 
indicators, television transmission from the vehicle, calls on the radio from the chase pilot, and space- 
positioning calls from the flight-test engineer. 

Simulation model. - For most of the program, the landing cues for the pilot in a HiMAT simulation 
included only the instruments, mapboards, and the ILS/glideslope error indicators. Although these are 
all valid cues, they could not achieve the same effect as the television transmission used in actual 
flight. During flight, as soon as the pilot can identify the runway, his scan focuses more on the 
television picture and less on the cockpit instruments. To help alleviate this lack of fidelity in the 
simulation, a display of the runways on the dry lakebed was developed on a recently purchased Evans 
and Sutherland Graphics System. 



HiMAT was actually flown using cockpit instruments, ILS/glideslope error indicators, television 
transmission from the vehicle, calls on the radio from the chase pilot, and space-positioning calls from 
the flight-test engineer. 

It was not flown using synthetic vision. 

The AUVSI Authors have reproduced a picture in their article with the caption, "The HiMAT RPV remote 
cockpit showing synthetic vision display. Photo courtesy of NASA." 

This picture is identical to the picture in Sarrafian Figure 5 f Ref. 11) , August 1984, PDF page 10} but the 
Sarrafian picture has a different caption. It says, " HiMAT simulation cockpit." 



The HiMAT RPV remote cockpit showing 
synthetic vision display. Photo courtesy of 
NASA. 




ECU 23757 
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The monitor shows a picture of the kind shown in Sarrafian Figure 8 or Figure 9 (along with a 
considerable amount of what appears to be reflected glare). The picture was produced by an Evans and 
Sutherland Picture System which requires a calligraphic monitor. 

Here's the thing. "The vehicle was flown with cockpit display instruments until the landing approach 
phase of the flight when the camera aboard the aircraft was activated to provide the pilot with a television 
display during the approach." 

In order to display the video from the camera aboard the aircraft, the Ground Cockpit that controlled the 
aircraft had to have a raster-scan monitor. 

Raster-scan monitors and Calligraphic monitors are incompatible. 

The picture shows the Simulation Cockpit, and the Simulation Cockpit could not be used to control the 
aircraft. 

Why did the AUVSI Authors change the caption? 
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Visuai-Proprioceptive Cue Conflicts in the Control of Remotely Piloted Vehicles, Reed. 1977 

In paragraph 9 the AUVSI Authors state: 

Also in 1979, the Air Force published research identifying human factors problems that would have 
to be overcome in RPV cockpit design ("Visual- Proprioceptive Cue Conflicts in the Control of 
Remotely Piloted Vehicles" by Reed in 1977). NASA would use this in the design of the HiMAT 
RPV 3D visual system in 1984. 



Ref. 14 provides the link to the Reed report. 
This is what the Reed report was about: 

1 . From page 5 (PDF page 8): 

An operator is asked to maneuver a remotely piloted vehicle (RPV) from an airborne control 
station (a mother ship). This station is equipped with a television monitor, control stick, and other controls 
and displays necessary to maneuver the RPV through a specified course. The RPV, containing a television 
camera mounted in its nose, relays an image of the terrain to be displayed on the television monitor in the 
control station. Thus, the visual scene displayed to the operator represents the scene viewed by the 
camera. The task of the operator is to use the controls and displays to "fly" the RPV in much the same 
way he would fly a conventional aircraft. 

The scenario is complicated by several factors. First, the visual inputs to the operator from the RPV are 
independent of the motion inputs from the control station. Thus, the operator will experience motion cues 
that are uncorrelated with the visual inputs received from the RPV. Second, while traditional pilot training 
programs operate on the philosophy that proprioceptive cues provided by the motion of the aircraft should 
be disregarded, research has shown that these cues are compelling, not easily ignored, and may improve 
performance when used in training simulators (see, for example, Borlace, 1967; Cohen, 1970; Douvillier, 
Turner, McLean, & Heinle, 1960; Fedderson, 1961; Huddleston & Rolfe, 1971; Rathert, Creer, & 
Douvillier, 1959; Ruocco, Vitale, & Benfari, 1965). The task simulated in the experiment presented here, 
however, required that the RPV operator disregard sensations of motion in order to maintain adequate 
performance. Under conditions of visual -proprioceptive conflict (as when the mother ship and/or the 
RPV are in turbulence) the stereotypic responses of pilots to correct angular accelerations will be 
inappropriate. 

2. From page 7 (PDF page 10): 

Visual system. The visual system consisted of a three-dimensional terrain model (a modified SMK-23 
Visual Simulator , The Singer Company), television camera and optical probe , and three monochromatic 
television monitors. The terrain model provided "real-world ground cues for visual tracking over the 
surface. The real-world to terrain model scale was 3,000:1 and represented a six by twelve-mile (9.65 by 
19.3 km) area. The model was mounted on an endless belt that was servo-driven to represent the 
continuous changes in scene as the simulated RPV traveled alone north-south directions. A television 
camera viewed the terrain model through an optical probe that contained a servoed mechanical assembly 
to permit the introductions of heading, roll, and pitch. Both the camera and probe were mounted on a 
servo-driven carriage system that moved across the terrain model to simulate movement of the RPV along 
east-west directions and in and out to simulate altitude changes. 
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The SMK-23 was also used in The Lunar Roving Vehicle (LRV) simulator (Rel 15} . This shows what an 
SMK-23 looks like. 
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The SMK-23 used a television camera with an optical probe to fly over the terrain model contained on a 
servo-driven endless belt. 

If Reed had had synthetic vision why would he have used the SMK-23 mechanical contraption? 



The only link between Reed and HiMAT is that the Hi MAT aircraft could be landed by either a ground- 
based pilot or an airborne controller (the backseat chase pilot in the TF-104G aircraft). \Ref 13 - Evans 
& Schilling, PDF page 9} 

Actual- The backup control system (BCS) is the second of the two independent flight control 
systems required for the Hi MAT program. The BCS control law is resident in one of the two 
onboard digital computers. The BCS is a full-authority, three-axis, multirate digital controller 
with stability augmentation functions and mode command functions (ref. 4). Each of seven 
modes is semiautomatic with the pilot providing direction by way of discrete command inputs. 
The BCS commands elevons for pitch and roll control and rudders for yaw control, and has an 
autothrottle for speed modulation. 
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The BCS was designed to provide well-controlled dynamics throughout the flight envelope, to 
have the ability to recover from extreme attitudes, and to bring the vehicle to a selected site and 
effect a successful landing by either a ground-based pilot or an airborne controller (the backseat 
chase pilot in the TF-104G aircraft) . It was designed to provide these features for an unstable 
vehicle configuration of no more than 10-percent aft mean aerodynamic chord center-of- gravity 
location. The original HiMAT BCS was developed by Teledyne Ryan Aeronautical for the 
onboard microprocessor computer, and was programmed entirely in Intel 8080 assembly 
language. 

While HiMAT might have used the results of the Reed report to select the airborne controller (the 
backseat chase pilot in the TF-104G aircraft) Reed did not use synthetic vision and neither did HiMAT. 
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Simulators 

The AUVSI Authors describe several flight simulators, such as the RC AeroChopper by Ambrosia 
Microcomputer Products [Paragraphs 15 and 16] and Bruce Artwick's "Flight Simulator" for the Apple II, 
which ultimately became Microsoft Flight Simulator. [Paragraph 5] 

RC AeroChopper was developed by David R. Stern at Ambrosia Microcomputer Products. The following 
is from an email correspondence with Mr. Stern: 

Question 1 : Did AeroChopper use a 3D terrain database? 

Mr. Stern: I guess it did, although the ground was a plane with 3D objects (and a 2D runway) 
scattered around (trees, pylon, towers with crossbar to fly under). 

Question 2: If so, did it represent real terrestrial terrain? 
Mr. Stern: No. 



Question 3: Did AeroChopper do real 3D? 

Mr. Stern: Yes. All the objects including the aircraft were described by a list of points, a list of point 
pairs for lines and a list of which points were in each polygon, each point had an x,y and z 
component. The original version was started in 1984, shown at the first R/C show (I think in Storm 
Lake Iowa) in the summer of 1986, had only vector graphics. About 1990 1 changed to filled 
polygons. The aircraft was rotated (pitch, yaw and roll) slightly each frame with respect to the fixed 
coordinate system. Then the aircraft and all background objects were rotated and scaled depending on 
the relative position of the "camera". 

The view on the screen was initially from a fixed point about eye level for a standing R/C pilot. The 
"camera" rotated to keep the aircraft on the screen. In the late 80s, I added two different viewpoint 
options ("camera" flying near the aircraft) . One mode was just behind the aircraft, looking in the 
direction the aircraft was pointed. The second camera mode followed the aircraft to keep it from 
getting too far away but slowed and stopped as the aircraft got closer. You can often see the ground 
objects from the air in these modes. 

I developed the first version on the Atari 520 ST computer in 68000 assembly language. Then I 
developed an Amiga version and then a Macintosh version. In about 1991, 1 developed an 80286 
version for a DOS machine. (The latest version requires a Windows 98 or older machine with an 
RS232 port and runs under DOS) 

RC AeroChopper was a significant achievement for the home computers available at the time and was a 
highly regarded simulator { Ret 17) but: 

1 . It did not use a digital elevation database; "... the ground was a plane with 3D objects (and a 2D 
runway) scattered around (trees, pylon, towers with crossbar to fly under)," and thus, did not 
represent real terrestrial terrain. 
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2. It did not provide a computer-generated image of the external scene topography from the 
perspective of the flight deck that is derived from aircraft attitude, high-precision navigation solution, 
and database of terrain , obstacles and relevant cultural features. 

It was not synthetic vision. It was a simulator. 



Now, let's discuss Microsoft Flight Simulator [Ret 18} : 

Flight Simulator 5.1 was released in 1995. Microsoft Flight Simulator did not start using 3D terrain until 
Flight Simulator 2000 Pro, released in late 1999. 

From Ref. 19 : 

GRAPHICS 

We now have another complete globe to fly around. With the new mesh style scenery we have real 
elevation points that make the surrounding terrain rise and fall like the real earth. We have no more 
flat areas that just pop up into place at the last minute during a landing approach! 

Even then, it is not clear if the terrain database represents real terrain or is made up. 

The article mentions the new GPS feature: 

737 Panel 

The 737-400 panel is very nicely done. Simple, yet effective. This is where FS2000 is not much 
different than FS98. However, the overall clarity, coloring, detailing and some new systems make it 
much better. We now have nice popups for the throttle quadrant, radio stack, compass and best of 
all the new GPS. 

The GPS is part of the simulated 737 control panel. There is no suggestion that a physical GPS unit can 
be connected to the program. 

A simulator is not synthetic vision. A simulator might do a good job simulating synthetic vision. It might 
even use a Digital Terrain Elevation Database representing real terrestrial terrain, but that does not make 
it synthetic vision. It is a simulator. If it does not control a physical aircraft it is not synthetic vision. 
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When Did NASA Start Working on Synthetic Vision? 

From Ref20 : 

NEWS RELEASE 

May 28, 1999 

Synthetic Vision Could Help General Aviation Pilots Steer Clear of Fatalities 

Hampton, Virginia — Research Triangle Institute and six companies are teaming up to develop 
revolutionary new general aviation cockpit displays to give pilots clear views of their surroundings 
in bad weather and darkness. 

The RTI Team includes Flight International, Inc., Newport News, Virginia, (a GA aircraft user) 
and Archangel Systems, Inc., Auburn, Alabama, who are committed to early commercialization 
and will make significant cost share contributions. The starting point for the new system is 
Archangel's TSO'd and STC'd Cockpit Display System. 

RTI also has teamed with Seagull Technology, Inc., Los Gatos, California (a GPS and 
attitude/heading reference system technology firm), Crew Systems, Inc., San Marcos, Texas, (a 
designer of low-cost head up displays), and Dubbs & Severino, Inc., Irvine, California (an 
award-winning terrain database design company). In addition, FLIR Systems, Inc., Portland, 
Oregon (an infrared instrument manufacturer) has agreed to evaluate the costs and benefits of 
existing weather penetrating sensor technology. 

Limited visibility is the greatest factor in most fatal aircraft accidents, according to the Aviation 
Safety Program at NASA's Langley Research Center in Hampton, VA. The RTI team is among 
six selected by NASA to develop different applications of Synthetic Vision. 

The RTI team will design, develop, and certify a Synthetic Vision system for general aviation 
aircraft. The purpose is to reduce or eliminate controlled flight into terrain caused by visibility- 
induced human error. 

Synthetic Vision is a display system that will offer pilots an electronic picture of what's outside 
their windows, no matter the weather or time of day. The system combines Global 
Positioning Satellite signals with terrain databases and graphical displays to draw three- 
dimensional moving scenes that will show pilots exactly what's outside. 

The NASA Aviation Safety Program envisions a system that incorporates multiple sources of 
data into cockpit displays. The displays would show hazardous terrain, air traffic, landing and 
approach patterns, runway surfaces and other obstacles that could affect an aircraft's flight. 

The NASA Aviation Safety Program is a partnership with the FAA, aircraft manufacturers, 
airlines and the Department of Defense. This partnership supports the national goal 
announced by President Clinton to reduce the fatal aircraft accident rate by 80 percent in 
10 years and by 90 percent over 25 years. 
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Research Triangle Institute is an independent, not-for-profit organization that conducts R&D 
and provides technical services to industry and government. With a staff of more than 
1,600 people, RTI is active in aerospace and many other fields of applied technology. RTI 
was created in 1958 as the centerpiece of North Carolina's Research Triangle Park, where its 
headquarters are located. RTI's Aerospace Technology Center in Hampton, Virginia, will carry 
out the Synthetic Vision project. 

In a separate press release dated May 13, 1999 NASA announced {from Ret 21) : 

Industry teams submitted 27 proposals in four categories: commercial transports and business 
jets, general aviation aircraft, database development and enabling technologies. NASA 
and researchers from the Federal Aviation Administration and Department of Defense evaluated 
the proposals' technical merit, cost and feasibility. 

NASA has committed $5.2 million that will be matched by $5.5 million in industry funds to 
advance Synthetic Vision projects over the next 18 months. More money is expected to be 
designated later to accelerate commercialization and make some systems available within four to 
six years. 

Among the team leaders selected for the first phase of the program are: Rockwell Collins, Inc., 
Cedar Rapids, IA; AvroTec, Inc., Portland, OR; Research Triangle Institute, Research Triangle 
Park, NC; Jeppesen-Sanderson, Inc., Englewood, CO; the Avionics Engineering Center of Ohio 
University, Athens, OH; and Rannoch Corporation, Alexandria, VA. 

Rockwell Collins, Inc. will receive funds to develop synthetic vision for airliners and business 
jets. The AvroTec, Inc. and Research Triangle Institute groups will use their awards 
to create technologies for a general -aviation synthetic vision system. A team led by Jeppesen- 
Sanderson, Inc. will receive funds to develop terrain database requirements and system 
approaches. The Avionics Engineering Center of Ohio University and Rannoch Corporation will 
use their awards to design specific component technologies for Synthetic Vision. 

When did NASA start working on Synthetic Vision? 
The answer is: 1999. 

When did NASA first use synthetic vision to control a UAV? 
It was in the X-38 project. 

From Ret 22 . "Virtual Cockpit Window" for a Windowless Aerospacecraft 
Wednesday, January 01 2003 

A software system processes navigational and sensory information in real time to generate a 
three- dimensional- appearing image of the external environment for viewing by crewmembers 
of a windowless aerospacecraft. The design of the particular aerospacecraft (the X-38) is such 
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that the addition of a real transparent cockpit window to the airframe would have resulted in 
unacceptably large increases in weight and cost. 

When exerting manual control, an aircrew needs to see terrain, obstructions, and other features 
around the aircraft in order to land safely. The X-38 is capable of automated landing, but even 
when this capability is utilized, the crew still needs to view the external environment: From the 
very beginning of the United States space program, crews have expressed profound dislike for 
windowless vehicles. The well-being of an aircrew is considerably promoted by a three- 
dimensional view of terrain and obstructions. The present software system was developed to 
satisfy the need for such a view. In conjunction with a computer and display equipment that 
weigh less than would a real transparent window, this software system thus provides a "virtual 
cockpit window." 

The key problem in the development of this software system was to create a realistic three- 
dimensional perspective view that is updated in real time. The problem was solved by building 
upon a pre-existing commercial program — LandForm C3 — that combines the speed of flight- 
simulator software with the power of geographic-information-system software to generate real- 
time, three-dimensional -appearing displays of terrain and other features of flight environments. 
In the development of the present software, the pre-existing program was modified to enable it to 
utilize real-time information on the position and attitude of the aerospacecraft to generate a view 
of the external world as it would appear to a person looking out through a window in the 
aerospacecraft. The development included innovations in realistic horizon-limit modeling, three- 
dimensional stereographic display, and interfaces for utilization of data from inertial-navigation 
devices, Global Positioning System receivers, and laser rangefinders. Map and satellite imagery 
from the National Imagery and Mapping Agency can also be incorporated into displays. 

The Press Release from Rapid Imaging Software, Inc., which did the synthetic vision work for the X-38, 
states ( Ref. 23} 




On December 13th, 2001, Astronaut Ken Ham successfully flew the X-38 from a remote cockpit 
using LandForm VisualFlight as his primary situation awareness display in a flight test at 
Edwards Air Force Base, California. This simulates conditions of a real flight for the windowless 
spacecraft, which will eventually become NASA's Crew Return Vehicle for the ISS. We believe 
that this is the first test of a hybrid synthetic vision system which combines nose camera video 
with a LandForm synthetic vision display. Described by astronauts as "the best seat in the 
house", the system will ultimately make space travel safer by providing situation awareness 
during the landing phase of flight. 
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Other References cited by the AUVSI Authors 

"Pathwav-in-the-Sky Contact Analog Piloting Display " Knox and Leavitt, 1977 

In the article the AUVSI Authors state in Paragraph 7: 

In 1977, NASA researcher Charles Knox published "Pathway-in-the-Sky Contact Analog 
Piloting Display," which included a complete design for a synthetic vision system. It featured a 
computer that projected a 3D view of the terrain given an aircraft's position and orientation. This 
out-the-window perspective view was displayed on a CRT type display. Such displays were 
called "Pictorial Format" avionics systems, but we recognize them as containing all of the 
essential elements of a modern synthetic vision display. 

The pictures that will be reproduced shortly are from the Knox report {Charles E. Knox and John Leavitt). 
I have placed them with the descriptions from Knox pages 3-4. The complete Knox report is Ref. 24 . 

Everything comes together in Knox Figure 4, which shows the Airplane track-angle pointer and scale, the 
Airplane symbol with shadow superimposed, the Flight-path-angle scale, the Flight-path prediction 
vector, the Earth horizon, the Roll pointer, the Airplane altitude deviation from path, the Airplane flight- 
angle bars, the Programmed path-angle indicator, the Potential flight-path-angle box, and the 
Programmed flight path. 

The Programmed flight-path consists of two three-dimensional lines showing the predicted flight path of 
the airplane. Knox and Leavitt's work is significant but there is no terrain, there is no digital elevation 
database. There is no synthetic vision. 

From Knox Description of Path-in-the-Sky Contact Analog Piloting Display {Ref. 24} : 

Display Symbology 

The format of the PITS contact analog display shows airplane attitude information in the form of bank 
angle and pitch changes. Airplane performance information is shown in the form of airplane flight-path 
angle and flight-path acceleration (which may be used as thrust- or energy-management control). Both 
vertical and lateral path deviations during a tracking task are shown in pictorial form. 

Path-tracking situation information is shown through a combination of an airplane symbol, a vertical 
projection of the airplane symbol with an extended center line drawn at the altitude of the path, a flight- 
path predictor, and a drawing of the programed path (fig. 1). These four pieces of symbology are 
drawn in a perspective display format as if the observer's eye were located behind and above the 
airplane. 

The airplane symbol is a tetrahedron with a smaller tetrahedron at the tail to visually enhance pitch 
changes. The airplane's true position with respect to the path is at the symbol's apex. The symbol rolls 
and pitches about its apex in accord with the real airplane's attitude. 
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Altitude deviations from the programed path are indicated to the pilot pictorially by a vertical projection 
of the airplane symbol. The projection, drawn with dashed lines, may be thought of as a shadow; as 
shown in figure 2, it remains directly above or below the airplane at the altitude of the path. If the 
airplane is above the programed path, the shadow appears to be below the airplane symbol. If the 
airplane is below the programed path, the shadow appears to be above the airplane symbol. 




Altitude deviation 



(a) Above path. 




(b) Below path. 

Figure 2,~ Airplane symbol and aha Jew interaction Curing altitude deviations. 
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Since the shadow is always drawn directly above or below the airplane symbol, the pilot may readily 
identify lateral tracking deviations when they are combined with a vertical tracking error. Figure 3 
shows the perspective view of the shadow, the airplane symbol, and the path when the airplane is above 
and to the left of the path. 




Figure 3.- lirplane ibon red to Uft of path. 
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Altitude deviations from the programed path are also shown to the pilot in numerical form in a box in 
the upper right-hand comer of the display (fig. 4). The pilot is expected to use this information when the 
path and shadow are out of the display field of view, such as could occur during initial path captures. 

A flight-path prediction vector (fig. 4) in the horizontal plane is attached to the shadow. The prediction 
vector, indicated by a dashed line, shows the airplane's predicted path for the next 10 sec based on the 
airplane's present bank angle and ground speed. An extended shadow center line drawn from the apex 
of the shadow in the direction of the present track angle, is also shown to aid the pilot with the lateral 
tracking task. 
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Figure 5 shows the flight-path prediction vector and the present track indicator with the airplane in a 
left bank of 13°. 

c a □ Q o □ a ° ° 




Figur* 5.- The PITS display concept showing airplane below path 
olinblng in a left bank of 13°. 
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"The Electronic Terrain Map: A New Avionics Integrator". Small. P.M.. 1981 

In the article the AUVSI Authors state in Paragraph 8: 

In 1979, the U.S. Air Force completed its "Airborne Electronic Terrain Map Applications Study" 
and in 1981 published "The Electronic Terrain Map: A New Avionics Integrator" describing how 
a computerized terrain database could be displayed as an out-the-window 3D view allowing the 
pilot to "see" even at night and in other limited visibility situations. 

No, Small did not describe "how a computerized terrain database could be displayed as an out-the- 
window 3D view allowing the pilot to 'see' even at night and in other limited visibility situations." 

The Small report discusses the concept of a digital Electronic Terrain Map (ETM) and proposes that it be 
used for: 

1. Navigation; 

2. Terrain Following/Terrain Avoidance (TF/TA); 

3. Threat avoidance, analysis, warning, and display; 

4. Terrain Masking; 

5. Weapon delivery; 

6. Route planning. 

He does say, "An electronic map subsystem can generate perspective scenes, which are essentially 
computer generated images of the surrounding area, and an electronic map should be much easier to 
interpret," but: 

1 , The statement must be understood according to the meaning it would have had at the time the 
article was written (circa 1981 ); and 

2. Wishing for a desired result is not the same as teaching how to do it. 

This is what the Small report {Ret. 25} is about: 
From the section INTRODUCTION: 

INTRODUCTION 

Currently, the Air Force has in the inventory paper and film map systems , which were 
developed to support the high and level flight environment. These maps were an effective means 
of tapping the vast files of information stored in the Defense Mapping Agency (DMA) data base, 
when the crew had time to study and interpret them (in fact, much of their value was actually 
obtained from pre-flight mission preparations). Interviews with pilots indicate that paper maps 
are less useful for low altitude flights . Film maps with CRT annotation are somewhat better, but 
still have a fundamental limitation in that it takes an operator to access any information. That is, 
it is not possible to transfer information directly from the data base to any other avionics system 
when it is stored on paper or film maps in what is essentially an analog form. 

The map reading process is a demanding task that can be simplifie d by usin g a digital 
map subsystem which accesses the information needed and presents it in a form which can be 
easily interpreted. At low altitude, and with a line of sight limited to the next ridge line, it's very 
difficult to interpret standard paper maps, which are presented as a vertical projection of a large 
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area. An electronic map subsystem can generate perspective scenes, which are essentially 
computer generated images of the surrounding area , and an electronic map should be much 
easier to interpret. In addition, essential information from the map data base can be placed on the 
pilots Head Up Display, reducing the need for head down operations. 

Paper maps are clumsy to use, whether you are flying an aircraft or driving a car. An electronic map, if 
properly done, would make using a map easier. 

However, whether the map is electronic or on paper, you still have to know where you are. Small has not 
addressed that issue in this section. 

The issue of what Small might mean by "perspective scenes" will be addressed later. 

From the section FUTURE AIRCRAFT SYSTEM: 

FUTURE AIRCRAFT SYSTEM 

The purpose of adding an ETM subsystem to a future avionics suite is to provide map 
data and displays that can be interfaced with other subsystems to improve the performance of the 
terrain following/terrain avoidance (TF/TA) , threat avoidance and navigation avionics 
subsystems. The requirement for the simultaneous exchange of processed map data by three or 
four avionics subsystems will be the most difficult objective and important feature of the ETM. 
Development and incorporation of the advanced ETM concepts and technologies will be required 
to augment future threat avoidance, navigation, TF/TA, and weapon delivery avionics 
subsystems . Applications/examples of using these ETM concepts and/or technologies and the 
utilization of an ETM subsystem as a source of information follows. 



TF/TA 

The first example will be the automatic TF/TA avionics subsystem. Our existing 
automatic TF subsystems operate using only active sensors as sources of terrain profile 
information (i.e. radar) . This makes the subsystem totally dependent on the limitations of this 
single information source. In case of radar, range is limited to line of sight. Absolutely no 
information is available beyond line of sight. This forces the TF subsystem to provide 
unnecessarily large clearances over ridges to avoid the following peak which may or may not be 
imminent. Further, the TF subsystem must radiate on an almost continuous basis to provide a 
continuous terrain profile . Consequently detection and jamming are TF subsystem 
vulnerabilities. A digital terrain map could provide a second source of information to the TF 
flight command processing subsystem and the use of the map could serve as a backup in case of 
radar failures or jamming . The ETM could provide information concerning beyond line of sight 
conditions, enlarge the total field of view scanned for turning, and avoid the reduction of the duty 
cycle of the radar emission. In fact, this ability to scan the terrain to the side without turning and 
looking beyond the line of sight makes it possible for the first time to consider true automation of 
the TA function. Because of limitations in the existing DMA data base, the approach should be 
cautious and an active sensor will be needed to make absolute clearance measurements. None the 
less, the application of stored data, to the TF/TA problem can potentially have tremendous 
impact on Air Force capabilities in the low altitude flight mission. 
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1 . Existing Terrain Following systems use active radar to profile the terrain. The radar is line-of-sight, so it 
cannot see farther terrain hidden by closer terrain. 

2. An Electronic Terrain Map would allow you to determine what is over the next ridge. However, "Because 
of limitations in the existing DMA data base, the approach should be cautious and an active sensor will 
be needed to make absolute clearance measurements. 

You still need to know where you are so you can locate your position on the map. 



THREAT AVOIDANCE 

The second example will be the threat avoidance avionics subsystem. The whole purpose 
of low altitude missions is to reduce the probability of detection and attrition. If the threat 
avoidance problem is solved without regard to the location and lethal range of threats, the 
resultant path may place the aircraft in greater jeopardy than before. Terrain masking and launch 
dynamics limitations must be exploited to the fullest. Careful selection of the aircraft's routes to 
the target may be done by the crew or automatically. In either case, a digital map is required to 
provide the terrain information and the position of the threats identified by the avionics system. 
Pre-mission planning can provide a starting point for this analysis, but the dynamics of the threat 
assessment makes it essential that the crew be able to redefine the mission as new information is 
received from command and control functions or via the aircraft's own suite of threat defense 
sensors. 

1 . If you have a good terrain map you can use the terrain to hide your aircraft from those whom you do 
not want to know where you are or if you are even in the area. 

2. If your terrain map shows you where the threats are, don't go there. 
You still have to know your map position. 



NAVIGATION 

The third example will be the navigation avionics subsystem. With the addition of a 
correlator to the avionic suite and using the on-board sensors together with the ETM, navigation 
can be accomplished. Also, by displaying the ridge lines derived from stored terrain data on the 
head up display, passive navigation is possible. Hence, the ETM could also improve the 
utilization of the navigation subsystem. 

Small does not say what he means by a "correlator" or which onboard sensors he would use them with. 
There can be several types of "correlators." 

1 . You can visually look out your aircraft window at the terrain (mountains, lakes, rivers) and cultural 
features (towers, highways) and then look at a map and try to find them. Then you figure out where you 
would be on the map to see what you are seeing. The map can be paper or electronic. An example of a 
paper map converted to digital format is in Ref 26 . This is part of the Washington Sectional Aeronautical 
Chart, Scale 1 :500,000 55th Edition, published March 3, 1994 by U.S. Department of Commerce 
National Oceanic and Atmospheric Administration National Ocean Service. Click Here for map PDF . If 
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you are not familiar with the symbology used in paper sectional maps here is the Washington Legend . 

If you use the Zoom and Pan features of Acrobat you will see the advantages of an electronic version of 
a paper map (i.e., a digital map). 

2. You can use a computer to do the correlation, such as the method taught by Horn and Bachman in 
Using Synthetic Images to Register Real Images with Surface Models. {Ref. 27} 

Abstract: A number of image analysis tasks can benefit from registration of the image with a model 
of the surface being imaged. Automatic navigation using visible light or radar images requires exact 
alignment of such images with digital terrain models. In addition, automatic classification of terrain, 
using satellite imagery, requires such alignment to deal correctly with the effects of varying sun 
angle and surface slope. Even inspection techniques for certain industrial parts may be improved by 
this means. 

Small has not mentioned Terrain Referenced Navigation. In Terrain Referenced Navigation a Radar or 
Lidar is used to take a few elevation measurements of the terrain. These measurements are matched to 
the terrain in a digital terrain elevation database. 

An early example of Terrain Referenced Navigation is U.S. Patent 3,328,795 Fixtaking Means and 
Method issued June 27. 1 967 to Hallmark, {Ref 28} From Column 2, lines 18-53: 

Previously proposed fixtaking and navigational systems have sought to utilize terrain elevation 
data, and they have been based upon the analog comparison of sample data which are the 
continuous, analog representation of continuous variations in terrain elevations, with similar 
data contained in contour maps employed as such. At least some of the sample and known data 
hence have always been graphically or photographically displayed on actual sheets of paper, 
rectangles of photographic film, etc., and the values represented thereby have been shown as 
physically measurable along at least two axes. Because of the nature of the data employed, 
cumbersome and unwieldly equipments for photographic development, superposition of map 
over map, orthogonal adjustments of one set of data relative to another, etc. have been 
unavoidable sources of added weight, complexity, error, and malfunction. 

The present invention does not employ continuously recorded, analog data, but has as one of its 
bases the use of quantized terrain altitude information taken at discrete points. A numerical 
comparison of sample and prerecorded data is performed at high speed, and with results 
predictable and repeatable for the same inputs, by a digital computer. Since the digital computer 
and associated components are relatively unaffected by noise, vibrations, nuclear radiation, etc., 
no equipment is required for performing two-dimensional data comparisons, and no feedback or 
nulling circuitry is needed for determining the point of best physical correlation of the sample 
with the pre-recorded data. As distinguished from systems utilizing analog information, the 
digital computer is free from the sources of error unavoidably present where analog comparisons 
are made and hence is not only more accurate but is able to tolerate relatively large errors in 
sample and known data values without compromising fixtaking accuracy. 

TERCOM (Terrain Contour Matching) uses contour matching instead of elevations. U.S. Patent 
4,347,511 Precision navigation apparatus issued August 31,1 982 to Hofmann , et al. (Ref. 29} 
mentions: 
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''Aviation Week & Space Technology", Feb. 25, 1974 , page 50, ff, discloses the Tercom process. In 
the latter, barometric measuring devices and radio altimeters produce altitude profiles during specific 
time intervals of a flight over characteristic terrain. The one-dimensional differential profile between 
the barometric altitude and altitude above ground is compared with a two-dimensional reference 
profile. Here, the measured altitude profile is adjusted until the best correlation is achieved, so that 
the exact position of the aircraft results. 

There are some problems with Terrain Referenced Navigation and Tercom: 

1 . They are not reliable if the terrain changes after the Digital Terrain Map is made. Terrain can 
change seasonally due to snow accumulations or permanently due to vegetation growth (trees) or 
new buildings (technically, a cultural feature). 

2. They do not work over large flat terrain. (See Ret 30} 

3. They do not work over bodies of water. 

Although Terrain Referenced Navigation and Tercom systems that use Radar or Lidar still send out 
signals that can be detected, the signals are far less detectable than the signals used in Small's 
description of TF/TA systems. Small's TF/TA system uses a radar to scan the terrain, which is why it 
cannot see beyond the next ridge. 

Small's omission of Terrain Referenced Navigation and Tercom is puzzling. 

Small gives a choice between Radar-scanned terrain and finding your location on a map using an 
undefined method of adding a correlator to the avionic suite and using the on-board sensors together 
with the Electronic Terrain Map (ETM). 



What did Small mean when he said, "An electronic map subsystem can generate perspective scenes, 
which are essentially computer generated images of the surrounding area, and an electronic map should 
be much easier to interpret?" 

In the 1980s {and well into the 1990s) the conventional wisdom was that Real 3D graphics was too 
computationally intensive to do in real time without large and very expensive hardware. 

Honeywell was the leader in avionics. Harris was probably a close second. They both spent the 1980s 
and 1990s competing with each other to see who could do the best fake 3D. 

For example, U.S. Patent 4,660,157 Real time video perspective digital map display method issued 
April 21 , 1987 to Beckwith, et al. {Ret 31} 

Instead of mathematically rotating the points from the database the '157 Patent accounts for the aircraft's 
heading by controlling the way the data is read out from the scene memory. Different heading angles 
result in the data being read from a different sequence of addresses. 

From Column 3, lines 21 - 38: 

The addresses of the elevation data read out of the scene memory representing points in the two- 
dimensional scene of the terrain are then transformed to relocate the points to positions where 
they would appear in a perspective scene of the terrain. Thus, each point in the two-dimensional 
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scene is transformed to its new location in the perspective scene to be displayed on the viewing 
screen , and in the process, the data is automatically oriented with a heading-up disposition. The 
transformed points are then stored in a speed buffer for further processing by sun angle and line 
writing logic prior to being stored in a display memory from which data is read out to the display 
screen. Since data in the display memory represents one-to-one data to be displayed on the CRT, 
this data will be referred to as pixels (picture elements) in terms of its storage in the display 
memory for transfer to the CRT display. 



The '1 57 patent accounts for the roll attitude of the aircraft by mathematically rotating the screen data 
after it is projected. From Column 12, lines 42 - 47: 

The points which are output by the perspective transform circuit 1 10 are supplied to a screen 
rotation circuit 120 which serves to rotate the display data in accordance with the roll of the 
aircraft so that the display will accurately depict the view as it would appear, if visible, through 
the window of the aircraft. 

Beckwith displays only points. 

Fake 3D + Only Points does not qualify as what is now considered synthetic vision. 



There is Honeywell's U.S. Patent 5,179,638 Method and apparatus for generating a texture mapped 
perspective view issued January 12, 1993 to Dawson, et al. (Ret. 32) 

It even has the word "perspective" in the title, but the perspective it produces is a trapezoidal perspective, 
not a real 3D projected perspective. 

Dawson '638 incorporates by reference a number of other patents and patent applications, and 
determining exactly what Dawson meant in '638 requires following a trail through these patents. The 
short version is that what Dawson means by "perspective" is contained in U.S. Patent 4,884,220 
Address Generation with Variable Scan Patterns issued November 28, 1989 to Dawson (again), {Ref. 
33} which is incorporated by reference by Dawson '638. 

After discussing the shortcomings of prior art, Dawson '220 says (Column 2, line 56 through Column 3, 
line 2): 

This invention differs from the prior methods of perspective view generation in that a trapezoidal 
scan pattern is used instead of the radial scan method. The trapezoidal pattern is generated by 
an orthographic projection of the truncated view volume onto the cache memory (terrain data). 
The radial scan concept is retained, but used for an intervisibility overlay instead of the 
perspective view generation. The radial scan is enhanced to include a full 360 degree arc with 
programmable attributes. The rectangular pattern retains the parallel scan methodology for plan 
view map generation. Both a nearest neighbor and a full bilinear interpolation method of scan 
address generation are implemented. 

And now we know what Dawson means by "perspective." 



A real 3D perspective is a 3D projection. 
Anything else is Fake 3D. 
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If you think Fake 3D is just as good as Real 3D then the next time someone owes you money tell them 
that it's ok to pay you in fake dollars. 

There is also the matter that Small is only wishing for a desired result. Wishing for a desired result is not 
the same as teaching how to do it. 

Not only did Small not teach it, he was not clear in saying what he was wishing for. 



VCASS: An Approach to Visual Simulation, Kocian, P., 1977 

In the article the AUVSI Authors state in Paragraph 6: 

This emergence of computer flight simulation in the 1970s appears to have sparked a 
monumental amount of research. The U.S. Air Force began its Visually Coupled Airborne 
Systems Simulator (VCASS) program, with a particular eye toward future-generation fighter 
aircraft ("VCASS: An Approach to Visual Simulation," Kocian, D., 1977). 

The Kocian report is available in Ret. 34 . 
Summary 

Kocian is about using a Helmut Mounted Display (HMD) with a Head Position Sensing System to replace 
large expensive hemispherical display systems used in simulators. The simulator is used to develop the 
visual interface used by crew members to control advanced weapon systems. This visual interface can 
then be used in airborne operations. 

During simulation a representative visual scene is generated by the graphics or sensor imagery 
generators but, from Paragraph 1 1 (emphasis added): 

For an airborne VCASS capability, it is only necessary to install the VCS components along 
with a small airborne general purpose computer in a suitable aircraft and interface a 
representative programmable symbol generator to an on-board attitude reference system in 
order to synthesize either airborne or ground targets . 

The airborne version does not synthesize a visual scene, so it is not synthetic vision. 
Details 

A Visually-Coupled System is one that visually couples the operator to the other system components 
through the use of a Helmut Mounted Display (HMD) and Helmut Position Sensor. From Paragraph 9: 

The key components of VCASS will be VCS hardware which includes the HMS and 
HMD. These components are used to "visually-couple" the operator to the other system 
components he is using. AMRL has pioneered efforts in the research, development and testing of 
these hardware techniques. 
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A system using a Helmut Mounted Display with a Helmut Position Sensor is less expensive than the 
hemispherical projection systems being used and produces better results. Paragraph numbers have 
been added to the following paragraphs from Kocian. 

[1] In recent years Air Force operational units have experienced a continuing trend 
downward in the number of flight hours in aircraft that can be provided to each individual pilot 
for training and maintaining proficiency. This comes at a time when aircraft systems are 
becoming ever more complex and sophisticated requiring comparatively more hours for training 
to maintain the same relative flying proficiency. With increasing costs for fuel and aircraft and 
the failure of DoD funding to keep pace with these costs, the trend is almost sure to continue. In 
adjusting to the realities of keeping overall experience at a satisfactory level and reducing costs, 
procurement of aircraft simulators has become a necessity. 

[2] The rapid proliferation of simulators with no standard technical criteria as a guide has 

resulted in the evolution of several different design approaches. Most existing visual scene 
simulators utilize electro-optical devices which project video imagery (generated from a sensor 
scan of a terrain board or a computer generated imagery capability) onto a hemispherical dome 
or set of large adjacent CRT displays arranged in optical mosaics with the weapon, vehicle, and 
threat dynamics being provided by additional computer capabilities. 

[3] These large fixed-base simulators suffer from the following drawbacks. The majority 

of the visual projection techniques used in these simulators do not incorporate infinity optics 
which provide collimated visual scenes to the operator. Those which do are large and expensive 
and incorporate large CRT displays. The luminance levels and resolution of these displays are 
usually low and do not represent true ambient conditions in the real environment. Additionally, 
hemispherical infinity optics are difficult to implement and this technique requires excessive 
computer capacity to generate imagery due to the need for refreshing an entire hemisphere 
instantaneously, regardless of where the crew member is looking. In this regard, existing 
computer capability is not used effectively to match the channel capacity of the human visual 
system. There are also generally no stereoscopic depth cues provided for outside of-cockpit 
scenes. Another important drawback to these simulators is that the visual simulation is not 
transferrable to the actual flight environment, i.e., the ground-based system cannot be transferred 
to an actual aircraft to determine simulation validity. Finally, most existing techniques are very 
expensive and do not allow the flexibility of incorporating other display design factors such as 
different head-up display image formats, fields-of-view (FOV), representative cockpit 
visibilities, and optional control and display interfaces. 

[4] A quite different approach to solving the visual presentation problems of aircraft 

simulators is to employ the use of visually coupled systems (VCS). For many years it has been 
the mission of the Aerospace Medical Research Laboratory to optimize the visual interface of 
crew members to advanced weapon systems. This mission has been primarily pursued in two 
areas: (1) the establishment of control/display engineering criteria; and (2) the prototyping of 
advanced concepts for control and display interface. An important part of fulfilling this mission 
has been the development of VCS components which includes head position sensing systems or 
helmet mounted sights (HMS), eye position sensing systems (EPS) and helmet mounted displays 
(HMD). 



A103 



Case 3:09-cv-00421 -LRH-VPC Document 1 -1 Filed 07/31/09 Page 1 03 of 1 28 



36 

During simulation a representative visual scene is generated by the graphics or sensor imagery 
generators. From Paragraph 7: 

A more detailed analysis of the problem has produced a set of characteristics which a more 
ideal aircraft simulator might possess. Of primary importance is that it should be a flexible visual 
scene simulation providing synthesized out-of-the-cockpit visual scenes and targets , a 
representative vehicle whose type can be altered, threat and weapon dynamics, flexibility of 
control and display configurations, and inputs from sensor or real world imagery. It should be 
portable if possible and provide alternatives for crew station display options including number 
and configuration. This simulator should also be useable in both simulated air-to-ground weapon 
delivery and air-to-air engagement scenarios. Finally, it should be possible to use the same 
system in ground fixed base and motion base simulators as well as in aircraft. 



However, the airborne version does not synthesize the out-the-cockpit visual scene. It only displays the 
symbols used in its role as a weapons controller. That is why the airborne version only needs a small 
airborne general purpose computer. From Paragraph 11 (emphasis added): 

For an airborne VCASS capability, it is only necessary to install the VCS components 
along with a small airborne general purpose computer in a suitable aircraft and interface a 
representative programmable symbol generator to an on-board attitude reference system 
in order to synthesize either airborne or ground targets . This approach has the ultimate 
flexibility of utilizing the same symbol set , threat dynamics, etc., in the air that were originally 
used in the ground simulation. In either case, the crew member will engage electronic targets 
(either air-to-air or air-to-ground) and launch electronic weapons. His performance in these tasks 
in turn will be recorded and assessed for performance or utilized as training aids for the crew 
member or operator. 



The airborne version does not synthesize a visual scene, so it is not synthetic vision as the term is now 
used. 

In addition, the Kocian report describes a work-in-progress. From Paragraph 19: 

The design considerations involved in building a helmet-mounted display for the 
VCASS simulation present a more formidable and subjective set of problems whose solution is 
not entirely clear . It is certain that a larger display field-of-view is required but how large 
remains an unanswered question. The optical physics that are part of the display design imposed 
constraints which are difficult to resolve. Currently, an interim display possessing a 60 degree 
instantaneous field-of-view is planned for the VCASS; however, recent studies have shown that 
this may not be large enough especially when viewed with one eye. This leads naturally to 
biocular or binocular configurations. A whole host of human factors problems then becomes 
important including brightness disparity, display registration, and eye dominance. The decision 
whether or not to include color also becomes a major design decision not only because of the 
engineering development required but because user acceptance may weigh heavily on this factor. 



(The question whether or not to use color was later settled. The answer was color.) 
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U.S. Patent 5,566,073 Pilot Aid Using A Synthetic Environment 

issued October 15, 1996 to Margolin 



This patent was not mentioned by the AUVSI Authors. 



Abstract 

A pilot aid using synthetic reality consists of a way to determine the aircraft's position and 
attitude such as by the global positioning system (GPS), a digital data base containing three- 
dimensional polygon data for terrain and manmade structures, a computer, and a display. The 
computer uses the aircraft's position and attitude to look up the terrain and manmade structure 
data in the data base and by using standard computer graphics methods creates a projected three- 
dimensional scene on a cockpit display. This presents the pilot with a synthesized view of the 
world regardless of the actual visibility. A second embodiment uses a head-mounted display with 
a head position sensor to provide the pilot with a synthesized view of the world that responds to 
where he or she is looking and which is not blocked by the cockpit or other aircraft structures. A 
third embodiment allows the pilot to preview the route ahead or to replay previous flights. 

It teaches what is now known as synthetic vision in sufficient detail that it may be practiced by a Person 
having Ordinary Skill In The Art without undue experimentation. A Person having Ordinary Skill In The 
Art (POSIT A) is a legal term that is often fought over during patent litigation. 

This patent is a continuation of Application Ser. No. 08/274,394, filed Jul. 11, 1994, which is its filing 
priority date. The earliest known description of the invention is in Ret 35 . 

For those unfamiliar with Patent Law, the Claims are the legal definition of the invention. The purpose of 
the Abstract is to provide search terms only. 

See Ret 36 for the patent. (I am the inventor named in the patent.) 



A105 



Case 3:09-cv-00421 -LRH-VPC Document 1 -1 Filed 07/31/09 Page 1 05 of 1 28 



38 

U.S. Patent 5,904,724 Method and apparatus for remotely piloting an aircraft 

issued May 18, 1999 to Margolin 



This patent was also not mentioned by the AUVSI Authors. 



Abstract 

A method and apparatus that allows a remote aircraft to be controlled by a remotely located pilot 
who is presented with a synthesized three-dimensional projected view representing the 
environment around the remote aircraft. According to one aspect of the invention, a remote 
aircraft transmits its three-dimensional position and orientation to a remote pilot station. The 
remote pilot station applies this information to a digital database containing a three dimensional 
description of the environment around the remote aircraft to present the remote pilot with a three 
dimensional projected view of this environment. The remote pilot reacts to this view and 
interacts with the pilot controls, whose signals are transmitted back to the remote aircraft. In 
addition, the system compensates for the communications delay between the remote aircraft and 
the remote pilot station by controlling the sensitivity of the pilot controls. 

It teaches the use of synthetic vision (as the term is currently used) for remotely piloting an aircraft. It 
teaches it in sufficient detail that it may be practiced by a Person having Ordinary Skill In The Art without 
undue experimentation. 

This patent was filed January 19, 1996, which is its priority date. 

For those unfamiliar with Patent Law, the Claims are the legal definition of the invention. The purpose of 
the Abstract is to provide search terms only. 

See Ref. 37 for the patent. (I am the inventor named in the patent.) 



U.S. Patent Application Publication 20080033604 
System and Method For Safely Flying Unmanned Aerial Vehicles in Civilian Airspace 



In the interests of full disclosure I have the following patent application pending: U.S. Patent Application 
Publication 20080033604 System and Method For Safely Flying Unmanned Aerial Vehicles in 
Civilian Airspace. 

Abstract 

A system and method for safely flying an unmanned aerial vehicle (UAV), unmanned combat 
aerial vehicle (UCAV), or remotely piloted vehicle (RPV) in civilian airspace uses a remotely 
located pilot to control the aircraft using a synthetic vision system during at least selected phases 
of the flight such as during take-offs and landings. 

See Ref. 38 for the published patent application. (I am the inventor named in the application) 
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The Future of Synthetic Vision 



This is what the AUVSI Authors have said about synthetic vision [Paragraph 2]: 

More recently it has evolved away from being a piloting aid to a potentially powerful tool for 
sensor operators. 

and [Paragraph 22]: 

The recent availability of sophisticated UAS autopilots capable of autonomous flight control 
has fundamentally changed the paradigm of UAS operation, potentially reducing the usefulness 
of synthetic vision for supporting UAS piloting tasks. At the same time, research has 
demonstrated and quantified a substantial improvement in the efficiency of sensor operations 
through the use of synthetic vision sensor fusion technology. We expect this to continue to be an 
important technology for UAS operation. 

While I have no doubt that synthetic vision is very useful to the sensor operator, the news that its use in 
piloting UAVs is on its way out came as a big surprise to me. 

The AUVSI Authors have an ulterior motive in making the statements. Their real objective is to make 
people believe synthetic vision no longer has value in controlling Remotely Piloted Vehicles (aka UAVs) 
and that a Remotely Piloted Vehicle that is flown using an Autonomous control system is no longer a 
remotely piloted vehicle and therefore a sensor operator may use synthetic vision without infringing U.S. 
Patent 5,904,724. See Ref. 39 for the response Rapid Imaging Software's attorney sent to Optima 
Technology Group in 2006. 

The statements made by the AUVSI Authors form a distinction without a difference unless there is a wall 
between the sensor operator and the pilot that results in the sensor operator having no influence on how 
or where the UAV is flown. 

Consider the following scenarios: 

1 . The human sensor operator has synthetic vision; the human pilot does not. No communications is 
allowed between the human sensor operator and the human pilot lest the human sensor operator 
influence the human pilot where or how to fly the aircraft. Otherwise, it might be considered as 
contributing to piloting the aircraft. This results in a decidedly sub-optimal system. 

2. The human sensor operator has synthetic vision; the aircraft is flown autonomously (a machine pilot). 
No communications is allowed between the human sensor operator and the machine pilot lest the human 
sensor operator influence the machine pilot where or how to fly the aircraft. Otherwise, it might be 
considered as contributing to piloting the aircraft. This also results in a decidedly sub-optimal system. 

There are legal and political ramifications to this scenario. 

Someone has to be responsible for the operation and safety of the flight. The FAA defines "Pilot in 
Command" as {Ref. 5} : 

Pilot in command means the person who: 

(1 ) Has final authority and responsibility for the operation and safety of the flight; 

(2) Has been designated as pilot in command before or during the flight; and 
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(3) Holds the appropriate category, class, and type rating, if appropriate, for the conduct of the 
flight. 

It is unlikely that FAA will allow this responsibility to be delegated to a machine anytime soon. That's 
where the political ramifications come in. A UAV (especially a completely autonomous UAV) that injures 
or kills civilians would ignite a political firestorm that would ground the entire UAV fleet. 

Since there must be a human in the loop to be responsible for the operation and safety of the flight, that 
leaves a system where: 

1 . The human sensor operator has synthetic vision; 

2. The pilot is a machine; 

3. The operation and safety of the flight is held by a human (different from the sensor operator) who is 
designated the Pilot-in-Command; 

4. No communications is allowed between the human sensor operator and the machine pilot or the 
human sensor operator and the human Pilot-in-Command lest the human sensor operator influence the 
machine pilot or the human Pilot-in Command where or how to fly the aircraft. Otherwise, it might be 
considered as contributing to piloting the aircraft. This also results in a decidedly sub-optima! system. 

Frankly, it is stupid to cripple the utility of a UAV system in order to avoid paying a small patent licensing 
fee. Besides, the 724 patent is for the use of synthetic vision in a Remotely Piloted Aircraft. It is not 
limited to the use of synthetic vision by the crew member designated as the Pilot. 



An autonomous pilot would have to be really good. 

Even after 100 years of aviation, pilots still encounter situations and problems that have not been seen 
before. The way they deal with new situations and problems is to use their experience, judgment, and 
even intuition. Pilots have been remarkably successful in saving passengers and crew under extremely 
difficult conditions such as when parts of their aircraft fall off (the top of the fuselage peels off) or multiply- 
redundant critical controls fail (no rudder control). Computers cannot be programmed to display 
judgment. They can only be programmed to display judgment-like behavior under conditions that have 
already been anticipated. UAVs should not be allowed to fly over people's houses until they are at least 
smart enough to turn on their own fuel supply. 

[ On Apr. 25, 2006 the Predator UAV being used by the U.S. Customs and Border Protection agency to 
patrol the border crashed in Nogales, Ariz. According to the NTSB report (NTSB Identification 
CHI06MA121) when the remote pilot switched from one console to another the Predator was 
inadvertently commanded to shut off its fuel supply and "With no engine power, the UAV continued to 
descend below line-of-site communications and further attempts to re-establish contact with the UAV 
were not successful." In other words, the Predator crashed because the system did not warn the remote 
pilot he had turned off the fuel supply and it was not smart enough to turn its fuel supply back on. {Ref. 

40[] 

An autonomous UAV assumes the computer program has no bugs. 

Complex computer programs always have bugs no matter how brilliant or motivated the programmer(s). 
As an example, look at almost every computer program ever written. 

An autonomous Unmanned Combat Aerial Vehicle (UCAV) will have little chance against one flown by 
an experienced pilot using Synthetic Vision until Artificial Intelligence produces a sentient, conscious 
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Being. At that point, all bets will be off because a superior sentient artificial Being may decide that war is 
stupid and refuse to participate. It may also decide that humans are obsolete or are fit only to be its 
slaves. 

I propose yearly fly-offs: 

1 . A UCAV flown and fought autonomously against an F-22 (or F-35). 

2. A UCAV flown and fought by a human pilot using synthetic vision against an F-22 (or F-35). 

3. A UCAV flown and fought by a human pilot using synthetic vision against a UCAV flown and 
fought autonomously. 

And that is the future of Unmanned Aerial Systems. 
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FAA Title 14 Part 1 

The FAA definition of synthetic vision from: http ://ecf r.gpoaccess . gov/c gi/t/text/tex t- 

idx?c=ecfr&sid=4 1 b 1 c5 1 ea8cc4c9d 1 c5cbb94bbf28 1 38&rgn=div8&view-.text&node= 1 4: 1 .0. i . 1 . 1 .0. 1 . 1& 

idno=14 

Mirrored Copy: http://www.jmarEolin.com/svr/refs/ref05 faa.pdf 
Title 14: Aeronautics and Space 
PART 1— DEFINITIONS AND ABBREVIATIONS 

Synthetic vision means a computer-generated image of the external scene topography from the 
perspective of the flight deck that is derived from aircraft attitude , high-precision navigation 
solution , and database of terrain , obstacles and relevant cultural features. 

Synthetic vision system means an electronic means to display a synthetic vision image of the 
external scene topography to the flight crew. 

Reference 6 - FAA Synthetic Vision is based on the use of a Digital Elevation Database 
FAA SV Issues- Part 23 Position 

http://www.faa.gov/aircraft/air cert/design approvals/transport/media/Pt23 ApproachSlides.pdf 
Mirrored Copy: bttp://\vww.imargolin.com/svr/i-efs/ref06 Pt23ApproachSlides.pdf 

Federal Aviation Administration Part 23 Synthetic Vision Approval Approach 

Presentation to: FAA Synthetic Vision Workshop 
Name: Lowell Foster 
Date: Feb 14, 2006 

Page 13: 

SV Issues - Part 23 Position 

Terrain Data Confidence Issues Cont. 

• Complete database accuracy impossible to validate 

« Everyone gets their data from the same original source 

• Manufacturers are doing everything possible to verify the current data is accurate, but that is really 
just a confidence builder 

If accuracy of data base must be validated then SV is unapproveable. 

Page 14: 

PositionRisk Management / Mitigation of Terrain Uncertainties 

A111 



Case 3:09-cv-00421 -LRH-VPC Document 1 -1 Filed 07/31/09 Page 1 1 1 of 1 28 



44 

•No operational credit for SV -current minimums still apply 

•Significant safety benefits possible -outweighs what we consider minimal risk 

•Experience -large data base errors to date have been easy to recognize and report -very visible 
PFD and map display 

•Small data base errors such as an elevation point are likely to be insolated, so exposure to a 
misleading information situation is considered small 

• Current resolution tends to round-up the elevation data so that small errors are not as 
significant and on the conservative side 

Reference 7 - Digital Elevation Model: http://data.geocomm.com/dem/ 

Mirrored Copy: http://www.imargolin.com/svr/refs/ref07 usgs dem.pdf 

The USGS Digital Elevation Model (DEM) data files are digital representations of cartographic 
information in a raster form. DEMs consist of a sampled array of elevations for a number of 
ground positions at regularly spaced intervals. These digital cartographic/geographic data files 
are produced by the U.S. Geological Survey (USGS) as part of the National Mapping Program 
and are sold in 7.5-minute, 15-minute, 2-arc-second (also known as 30-minute), and 1 -degree 
units. The 7.5- and 15-minute DEMs are included in the large scale category while 2-arc-second 
DEMs fall within the intermediate scale category and 1-degree DEMs fall within the small scale 
category - (Source: USGS) 

Reference 8 - Digital Elevation Database improved by a Space Shuttle mission. 

http ://spacefli g ht.nasa.gov/shuttle/a rchives/sts-99/ 

Mirrored Copy: http ://www,j margoli n . com/s v r/ref s/refO 8 sts 99 , pdf 

STS-99 Crew Works in Shifts to Complete Mapping Mission 

Endeavour's international crew of seven spent 1 1 days in orbit during February 2000 mapping 
the Earth's surface with radar instruments. 



Space Shuttle Endeavour Maps the World in Three Dimensions 

The main objective of STS-99 was to obtain the most complete high-resolution digital 
topographic database of the Earth. 

The Shuttle Radar Topography Mission, or SRTM, was an international project spearheaded by 
the National Imagery and Mapping Agency and NASA, with participation of the German 
Aerospace Center, DLR. SRTM consisted of a specially modified radar system that flew onboard 
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the space shuttle during STS-99. This radar system gathered data that produced unrivaled 3-D 
images of the Earth's surface. 



A113 



Case 3:09-cv-00421 -LRH-VPC Document 1 -1 Filed 07/31/09 Page 1 1 3 of 1 28 



46 

Reference 9 - Honeywell IFPD Synthetic Vision System 

http://www.honeywell.com/sites/portal?page=ipfd primus&sniap=aerospace&theme=T5 
Mirrored Copy: http://www.imargolin.com/svr/refs/ref09 honeywell.pdf 




Reference 10 - NASA description of the HiMAT project: 
http://vvww.nasa.gov/centers/dryden/news/FactSheets/FS-025-DFRC.html 
Mirrored Copy: http://www.imargolin.com/svr/refs/reflO nasa himat.pdf 



Referenc e 11 - Simulator Evaluation of a Remotely Piloted Vehicle Lateral Landing Task Using a 
Visual Display, Shahan K. Sarrafian 

NASA Technical Memorandum 84916 (May 1984): 
http://www.nasa.gov/centers/dryden/pdf/87968main 1 205.pdf 
http://www.jiTiargolin.com/svr/refs/refl la sarrafian.pdf 

NASA Technical Memorandum 85903 (August 1984): 
htt p://www.nasa.gov/centers/dryden/pdf/87986main H-1246.pdf 
http://www.imargolin.com/svr/refs/ref1 lb sarrafian.pdf 

I converted this article to text in order to make it easier to search and to quote from. 
http://www.imargolin.com/svr/refs/refl lc sarrafian.doc . The downloaded PDF Fde is the controlling 
version. 
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Reference 12 - NASA Aviation Navigation 

Tutorial: http://virtuaIskies.arc.nasa.Eov/navigation/tutoriat/tutorial3.html 
Mirrored copy: http://www.imargolin.com/sviyrefs/ref~12 nasa ils.p df 

Reference 13 - THE ROLE OF SIMULATION IN THE DEVELOPMENT AND FLIGHT TEST 
OF THE HIMAT VEHICLE , M. B. Evans and L. J. Schilling, NASA-TM-84912, April 1984 

http://www.nasa.gov/centers/dryden / pdf/87962main H 1 1 90.pdf 

Mirrored Copy: http://www.imarKolin.com/svr/refs/refl 3 evans schilling.pdf 
From PDF page 13: 

Visual Landing Aid 

Actual. - Cues to the pilot during landing included the cockpit instruments, ILS/glideslope error 
indicators, television transmission from the vehicle, calls on the radio from the chase pilot, and space- 
positioning calls from the flight-test engineer. 

Simulation model. - For most of the program, the landing cues for the pilot in a HiMAT simulation 
included only the instruments, mapboards, and the ILS/glideslope error indicators. Although these are 
all valid cues, they could not achieve the same effect as the television transmission used in actual 
flight. During flight, as soon as the pilot can identify the runway, his scan focuses more on the 
television picture and less on the cockpit instruments. To help alleviate this lack of fidelity in the 
simulation, a display of the runways on the dry lakebed was developed on a recently purchased Evans 
and Sutherland Graphics System. 



Reference 14 - Visual-Proprioceptive Cue Conflicts in the Control of Remotely Piloted Vehicles, 
Reed, 1977, AFHRL-TR-77-57 

http://www.dtic.mil/src h/doc?collection=t2&id-ADAQ497Q6 
http://handle.dtic.mi1/10Q.2/ADAQ497Q6 

Mirrored Copy: http://www.jinargolin.com/s vr/refs/refl4 reed.pdf 



Page 5 (PDF page 8): 



VISUAL PROPRIOCEPTIVE CUE CONFLICTS IN THE CONTROL OF REMOTELY PILOTED 
VEHICLES 

I. INTRODUCTION 

An investigation was made of operator tracking performance under conditions of visual 
proprioceptive conflict. (The term proprioception as used here refers to sensations arising from the 
receptors of the nonauditory labyrinth of the inner car and from muscles, tendons, and joints. Kinesthesis 
refers to sensations of movement arising from the receptors other than the nonauditory labyrinth.) The 
experimental scenario is described as follows: An operator is asked to maneuver a remotely piloted 
vehicle (RPV) from an airborne control station (a mother ship). This station is equipped with a television 
monitor, control stick, and other controls and displays necessary to maneuver the RPV through a specified 
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course. The RPV, containing a television camera mounted in its nose, relays an image of the terrain to be 
displayed on the television monitor in the control station. Thus, the visual scene displayed to the operator 
represents the scene viewed by the camera. The task of the operator is to use the controls and displays to 
"fly" the RPV in much the same way he would fly a conventional aircraft. 

The scenario is complicated by several factors. First, the visual inputs to the operator from the 
RPV are independent of the motion inputs from the control station. Thus, the operator will experience 
motion cues that are uncorrected with the visual inputs received from the RPV. Second, while traditional 
pilot training programs operate on the philosophy that proprioceptive cues provided by the motion of the 
aircraft should be disregarded, research has shown that these cues are compelling, not easily ignored, and 
may improve performance when used in training simulators (see, for example, Borlace, 1967; Cohen, 
1970; Douvillier, Turner, McLean, & Heinle, 1960; Fedderson, 1961 ; Huddleston & Rolfe, 1 97 1 ; Rathert, 
Creer, & Douvillier, 1959; Ruocco, Vitale, & Benfari, 1965). The task simulated in the experiment 
presented here, however, required that the RPV operator disregard sensations of motion in order to 
maintain adequate performance. Under conditions of visual -proprioceptive conflict (as when the mother 
ship and/or the RPV are in turbulence) the stereotypic responses of pilots to correct angular accelerations 
will be inappropriate. 

The objectives of the experiment were to obtain data applicable to the following. 

1. The relative difficulty of controlling an RPV from an airborne station under different visual-motion 
combinations (e.g., visual-motion combinations that produce conflict, or no conflict). 

2. The relative ability of pilots, navigators, and nonrated Air Force officers to operate an RPV from 
an airborne station (i.e., the effect of previous experience). 

3. The differential effects of experience on the acquisition of skills necessary to operate an RPV. 

4. Selection and training of potential RPV operators. 

5. The need for motion in RPV training simulators. 



II. METHOD 

Simulation System 

This research utilized the Simulation and Training Advanced Research System (STARS) facility 
of the Advanced Systems Division, Air Force Human Resources Laboratory, Wright Patterson Air Force 
Base, Ohio. The equipment consisted of an operator station mounted on a motion platform, hydraulic 
pump, terrain model, television camera and optical probe, experimenter station, and a Sigma 5 digital 
computer. A brief description of the hardware system is presented as follows. 

Operator station. The operator station, illustrated in Figure 1, was designed to simulate the 
environment of an airborne control station. This station contained a television monitor that provided 
visual images relayed to h from a simulated RPV. These visual images were generated from a television 
camera and optical probe, which viewed the terrain model. The path followed by the camera and probe 
over the terrain model was commensurate with the vehicle flight path as determined by control stick 
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inputs provided by the subject. Since the control stick and visual system were independent of the motion 
platform, the capability existed for the subject to 

5 

[Figure 1. Operator station mounted on motion platform, {not usable}] 

maneuver the simulated RPV under various environmental conditions. This arrangement permitted the 
introduction of conditions in which the RPV alone, the airborne station alone, or both, were under air 
turbulence. 

The subject sat in an aircraft-type seat directly facing a 14- by 1 1 -inch (35.6 by 27.9 cm) 
television monitor, which was mounted in a center sectional panel of the operator console. The distance 
between the subject's eyes and the center of the television screen was 28 inches (71.1 cm). The viewing 
angle subtended 28.07° in the lateral plane and 22.23° in the vertical plane of the monitor. An altimiter, 
altitude warning light, and an attitude director indicator (ADI) were mounted on a flat sectional panel to 
the left of the subject and at an angle of 45° from the center panel (See Figure 2). The altimeter was a 
vertical straight- scaled indicator with a moving pointer that provided altitude readings in feet above sea 
level. An amber altitude warning light flashed whenever the simulated RPV altitude dropped to a level, 
below 180 feet (54.9 m), remained on whenever altitude exceed 1,000 feet (304.8 m) and was off between 
180 and 1,000 feet. 

A 6-inch (15.2 cm) side-arm rate control stick was mounted on the right-hand side console armrest 
(see Figure 2). The control was a spring-centered stick with a dual-axis (fee positioning) capability that 
required 4 ounces, (1 13.4 g) breakout force. The same amount of force was needed to hold the stick at full 
deflection. The range of deflection on both lateral (right - left) and longitudinal (fore - aft) stick was to 
25° (henceforth referred to as to 100 percent deflection). 

In addition, the operator station contained a foot switch to allow the subject to communicate with 
the experimenters. White noise was input to the subject's headset to mask external disturbances. The 
aircraft seat was equipped with a standard harness and lapbelt to protect the subject. An air conditioner 
maintained the station at 70° F (21. 1° C). Finally, incident illumination was at an average of .37 
footcandles at eye level. 

6 

[Figure 2. Operator station instruments and control stick, {not usable}] 

Motion system. The operator station was mounted on a motion platform that provided onset cues 
in two degrees of freedom of angular acceleration. Roll onset cues were provided by tilting the simulator 
about the longitudinal axis (i.e., the X axis) and pitch onset cues were provided by tilting the simulator 
about the lateral axis (i.e., the 'Y' axis). Motion was achieved by actuation of hydraulic cylinders mounted 
under the 9- by 8-feet (2.74 by 2.4 m) simulator platform, as shown in Figure 1. 

Visual system. The visual system consisted of a three-dimensional terrain model (a modified 
SMK-23 Visual Simulator, The Singer Company), television camera and optical probe, and three 
monochromatic television monitors. The terrain model provided "real-world ground cues for visual 
tracking over the surface. The real-world to terrain model scale was 3,000: 1 and represented a six by 
twelve-mile (9.65 by 19.3 km) area. The model was mounted on an endless belt that was servo-driven to 
represent the continuous changes in scene as the simulated RPV traveled along north-south directions. A 
television camera viewed the terrain model through an optical probe that contained a servoed mechanical 
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assembly to permit the introductions of heading, roll, and pitch. Both the camera and probe were mounted 
on a servo-driven carriage system that moved across the terrain model to simulate movement of the RPV 
along east-west directions and in and out to simulate altitude changes. The field of view represented on 
the television monitor subtended a viewing angle of 50° horizontally and 38° vertically over the terrain 
model. One television monitor was mounted in the operator station and the other two were located in the 
experimenter station. All three monitors had a 1,000-line resolution vertically. 

Experimenter station. The experimenter station contained the equipment necessary to monitor the 
status of the hardware/software and control activities of the subject, and to setup the various stimulus 
conditions. This station was manned by two experimenters. The task of the first was to prepare the system 
for operation, insure that all hardware was operating effectively and reliably prior and during the 
experiment, and set up the conditions for all experimental trials in accordance with a prepared check list. 
The task of the second experimenter was to determine the appropriate time for introducing specific stimuli 
to the subject. When certain criteria were met, the experimenter pressed a discrete hand-held insert button 
to initiate a stimulus trial. 

Computer system and interfaces. A Sigma 5 digital computer was used to drive the peripkeral 
equipment, and to record data during experimental runs. Resident software consisted of a real-time 
aerodynamic mathematical model, executive routine, and data recording programs. The 

7 



Reference 15 - Lunar Driving Simulator History 

http://www.knology.net/~skeetv/SimHist3.html 

Mirrored copy: http://www.imargolin.com/svr/refs/refl5 lunar driving history.pdf 
Reference 16 - Evans & Sutherland Picture System: 

Short Brochure: http://www.computerhistory.org/brochures/companies.php?alpha=d-f&company=com- 
42b9d8b7f4191 " ~~ ' " 

Full Brochure: 

http://arch ive.cpm pu terriistory.org /resources/text/Evans Sutherland/EvansSutherland,3D. 1 974. 1 0264628 
8j2df 

Mirrored copy: 

Short Brochure: http://www.imargolin.com/svr/refs/refl6 esps s.pdf 
Full Brochure: hUp;//ww w.imargolin.cp m/syr/refs/refl6 esps f.pdf 
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Reference 17 - RC AeroChopper Review: http://www.atarimagazines.com/startv3n9/fcaerochopper.htmI 
Mirrored Copy: http://www.iinargolin.com/svr/refs/refl7 aerochopper.pdf 

Reference 18 - Microsoft Flight Simulator 

Microsoft Flight Simulator 5.1 Screen Shot: 
http://www.imargolin.com/svr/refs/refl8 fs5 I screenshot.pdf 

Microsoft Flight Simulator History: http://www.imargolin.com/svi-/refs/refl8 fs history.pdf 

Reference 19 - Microsoft Flight Simulator's first use of terrain points: 
htt p://www.flightsim.com/ cgi/kds?$ -main/ review/fs2 QQ0.htm 

Mirrored copy: http://www.jmargolin.com/svr/refs/refl9_fs._first.pdf 

Reference 20 - News releases from RTI (Research Triangle Institute), Avidyne, AvroTec, and NASA 
announcing NASA had selected those companies to develop a synthetic vision system for General 
Aviation, www.jmargolin.com/svr/refs/ref20 nasal 999.pdf 

Reference 21 : NASA press release, May 13, 1999, http://quest.nasa.gov/aero/news/05- 1 3-99.txt 
Mirrored copy: http://www.imargolin.com/svr/refs/ref21 nasa pr.pdf 
Michael Braukus 

Headquarters, Washington, DC May 1 3, 1 999 

(Phone: 202/358-1979) 

Kathy Barnstorff 

Langley Research Center, Hampton, VA 
(Phone: 757/864-9886) 

RELEASE: 99-59 

SYNTHETIC VISION COULD HELP PILOTS STEER CLEAR OF FATALITIES 

NASA and industry are developing revolutionary cockpit 
displays to give airplane crews clear views of their surroundings 
in bad weather and darkness, which could help prevent deadly 
aviation accidents. 

Limited visibility is the greatest factor in most fatal 
aircraft accidents, said Michael Lewis, director of the Aviation 
Safety Program at NASA's Langley Research Center in Hampton, VA. 
NASA has selected six industry teams to create Synthetic Vision, 
a virtual-reality display system for cockpits, offering pilots an 
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electronic picture of what's outside their windows, no matter the 
weather or time of day. 

"With Global Positioning Satellite signals, pilots now can 
know exactly where they are," said Lewis. "Add super-accurate 
terrain databases and graphical displays and we can draw three- 
dimensional moving scenes that will show pilots exactly what's 
outside. The type of accidents that happen in poor visibility 
just don't happen when pilots can see the terrain hazards ahead." 

The NASA Aviation Safety Program envisions a system that 
would use new and existing technologies to incorporate data into 
displays in aircraft cockpits. The displays would show hazardous 
terrain, air traffic, landing and approach patterns, runway 
surfaces and other obstacles that could affect an aircraft's 
flight. 

Industry teams submitted 27 proposals in four categories: 
commercial transports and business jets, general aviation 
aircraft, database development and enabling technologies. NASA 
and researchers from the Federal Aviation Administration and 
Department of Defense evaluated the proposals' technical merit, 
cost and feasibility. 

NASA has committed S5.2 million that will be matched by $5.5 
million in industry funds to advance Synthetic Vision projects 
over the next 18 months. More money is expected to be designated 
later to accelerate commercialization and make some systems 
available within four to six years. 

Among the team leaders selected for the first phase of the 
program are: Rockwell Collins, Inc., Cedar Rapids, IA; AvroTec, 
Inc., Portland, OR; Research Triangle Institute, Research Triangle 
Park, NC; Jeppesen-Sanderson, Inc., Englewood, CO; the Avionics 
Engineering Center of Ohio University, Athens, OH; and Rannoch 
Corporation, Alexandria, VA. 

Rockwell Collins, Inc. will receive funds to develop 
synthetic vision for airliners and business jets. The AvroTec, 
Inc. and Research Triangle Institute groups will use their awards 
to create technologies for a general-aviation synthetic vision 
system. A team led by Jeppesen-Sanderson, Inc. will receive funds 
to develop terrain database requirements and system approaches. 
The Avionics Engineering Center of Ohio University and Rannoch 
Corporation will use their awards to design specific component 
technologies for Synthetic Vision. 

The Aviation Safety Program is a partnership with the FAA, 
aircraft manufacturers, airlines and the Department of Defense. 
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This partnership supports the national goal announced by President 
Clinton to reduce the fatal aircraft accident rate by 80 percent 
in 10 years and by 90 percent over 25 years. 

Because of advances in the last 40 years, commercial 
airliners are already the safest of all major forms of 
transportation. But with an accident rate that has remained 
relatively constant in the last decade and air traffic expected to 
triple over the next 20 years, the U.S. government wants to 
prevent a projected rise in the number of aircraft accidents. 

For a complete list of industry teams please check the 
Internet at: 

http://oea.larc.nasa.gov/news_rels/1999/May99/99-025.html 
- end - 

Reference 22 - Virtual Cockpit Window" for a Windowless Aerospacecraft, NASA Tech Briefs. 
January 2003, page 40. http://www.nasatech.com/Briefs/Jan03/MSC23096.html 
Mirrored Copy: http://www.imargolin.com/svr/refs/ref22 nasa techbriefs.pdf 

"Virtual Cockpit Window" for a Windowless Aerospacecraft 

Wednesday, January 01 2003 

A software system processes navigational and sensory information in real time to generate a 
three- dimensional- appearing image of the external environment for viewing by crewmembers 
of a windowless aerospacecraft. The design of the particular aerospacecraft (the X-38) is such 
that the addition of a real transparent cockpit window to the airframe would have resulted in 
unacceptably large increases in weight and cost. 

When exerting manual control, an aircrew needs to see terrain, obstructions, and other features 
around the aircraft in order to land safely. The X-38 is capable of automated landing, but even 
when this capability is utilized, the crew still needs to view the external environment: From the 
very beginning of the United States space program, crews have expressed profound dislike for 
windowless vehicles. The well-being of an aircrew is considerably promoted by a three- 
dimensional view of terrain and obstructions. The present software system was developed to 
satisfy the need for such a view. In conjunction with a computer and display equipment that 
weigh less than would a real transparent window, this software system thus provides a "virtual 
cockpit window." 

The key problem in the development of this software system was to create a realistic three- 
dimensional perspective view that is updated in real time. The problem was solved by building 
upon a pre-existing commercial program — LandForm C3 — that combines the speed of flight- 
simulator software with the power of geographic-information-system software to generate real- 
time, three-dimensional-appearing displays of terrain and other features of flight environments. 
In the development of the present software, the pre-existing program was modified to enable it to 
utilize real-time information on the position and attitude of the aerospacecraft to generate a view 
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of the external world as it would appear to a person looking out through a window in the 
aero spacecraft. The development included innovations in realistic horizon-limit modeling, three- 
dimensional stereographic display, and interfaces for utilization of data from inertial-navigation 
devices, Global Positioning System receivers, and laser rangefinders. Map and satellite imagery 
from the National Imagery and Mapping Agency can also be incorporated into displays. 

After further development, the present software system and the associated display equipment 
would be capable of providing a data-enriched view: In addition to terrain and obstacles as they 
would be seen through a cockpit window, the view could include flight paths, landing zones, 
aircraft in the vicinity, and unobstructed views of portions of the terrain that might otherwise be 
hidden from view. Hence, the system could also contribute to safety of flight and landing at night 
or under conditions of poor visibility. 

In recent tests, so precise was the software modeling that during the initial phases of the flight 
the software running on a monitor beside the video camera produced nearly identical views. 

This work was done by Michael F. Abernathy of Rapid Imaging Software, Inc., for Johnson 
Space Center. For further information, please contact Michael F. Abernathy, Rapid Imaging 
Software, Inc., 1318 Ridgecrest Place S.E., Albuquerque, NM 87108. MSC-23096. 



Reference 23 - Press Release from Rapid Imaging Software, Inc. 

( http://www.landform.com/pages/Pre ssReleases.htm) which states (near the bottom of the page): 
Mirrored copy: http://www.jmargolin.com/svr/refs/ref23 ris.pdf 




On December 13th, 2001, Astronaut Ken Ham successfully flew the X-38 from a remote cockpit 
using LandForm VisualFlight as his primary situation awareness display in a flight test at 
Edwards Air Force Base, California. This simulates conditions of a real flight for the windowless 
spacecraft, which will eventually become NASA's Crew Return Vehicle for the ISS. We believe 
that this is the first test of a hybrid synthetic vision system which combines nose camera video 
with a LandForm synthetic vision display. Described by astronauts as "the best seat in the 
house", the system will ultimately make space travel safer by providing situation awareness 
during the landing phase of flight. 

Reference 24 - Description of Path-in-the-Sky Contact Analog Piloting Display, Charles E. Knox 
and John Leavitt, October 1977, NASA Technical Memorandum 74057 

http://ntrs.nasa.gov/ai-chive/nasa/casi.ntrs.nasa.gov/l97800021 19 1 9780021 1 9.pdf 

Mirrored Copy: http://www.jmargolin.com/svr/refs/ref24 knox.pdf 
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Reference 25 - "The Electronic Terrain Map: A New Avionics Integrator", Small, D.M. USAF, Avionics 
Laboratory, Wright-Patterson AFB, OH, AIAA- 198 1-2289. In: Digital Avionics Systems Conference, 
4th, St. Louis, MO, November 17-19, 1981, Collection of Technical Papers. (A82-13451 03-04) New 
York, American Institute of Aeronautics and Astronautics, 1981, p. 356-359. 
http://www.jmargolin.com/svr/refs/ref25 small. pdf 

Converted to text using OCR: http://www.jmargo1in.com/svr/refs/ref25 small, html 



Reference 26 - This is part of the Washington Sectional Aeronautical Chart, Scale 1:500,000 55th 
Edition, published March 3, 1994 by U.S. Department of Commerce National Oceanic and Atmospheric 
Administration National Ocean Service. 

Map: http://www.jmargolin.com/svr/refs/ref26 pmapl .pdf 

Washington Legend showing paper map symbology: http://www.jmargoIin.com/svr/refs/ref26 pmap2.pdf 

Reference 27 - Using Synthetic Images to Register Real Images with Surface Models; Horn, Berthold 
K.P.; Bachman, Brett L. ; August 1977. 

MIT DSpace: http://hdl.handle.net/172Ll/5761 

Mirrored Copy: http://www.jmargolin.com/svr/refs/ref27 horn.pdf 

Abstract: A number of image analysis tasks can benefit from registration of the image with a model 
of the surface being imaged. Automatic navigation using visible light or radar images requires exact 
alignment of such images with digital terrain models. In addition, automatic classification of terrain, 
using satellite imagery, requires such alignment to deal correctly with the effects of varying sun angle 
and surface slope. Even inspection techniques for certain industrial parts may be improved by this 
means. 



Reference 28 - U.S. Patent 3,328,795 Fixtaking Means and Method issued June 27, 1967 to Hallmark. 

USPTO Database (Does not have htmp version): http : //patft.uspto. gov/ netacgi/nph- 
Parser?Sectl-Pr01&Sect2^Hll'OFF&d^PALL&p^l&u-%2Fnetahtml%2FPTO%2Fsrchnum.htm&r^l 
&f-G&l-50&s 1 -3.328,795.PN.&OS=PN/3,328,795&RS-PN/3.328.795 

PDF Version: http://www.jmargolin.com/svr/refs/ref28 3328795.pdf 

Reference 29 - U.S. Patent 4,347,51 1 Precision navigation apparatus issued August 31, 1982 to 
Hofmann, et al. 

From USPTO: http://patft.uspto.gov/netacgi/nph- 

Parser?SectUPT01&Sect2^HlTOFF&d=PALL&p=l&u^%2Hnetahtml%2FPTO%2Fsrchnum.htm&i-l 
&f=G&1=50&s 1=4.347.51 1.PN.&OS-PN/4.347.51 1&KS=PN/4,347,51 1 

PDF Version: http://www.jmargolin.com/svr/refs/re f29 434751 1 .pdf 
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Reference 30 - I don't know if Terrain Referenced Navigation works over Kansas, but I know Kansas is 
flat. From: http://www.guardian.co.uk/education/2QQ3/sep/25/research.highcreducation2 

This year, for instance, three geographers compared the flatness of Kansas to the flatness of a 
pancake. They used topographic data from a digital scale model prepared by the US Geological 
Survey, and they purchased a pancake from the International House of Pancakes. If perfect flatness 
were a value of 1.00, they reported, the calculated flatness of a pancake would be 0.957 "which is 
pretty flat, but far from perfectly flat". Kansas's flatness however turned out to be 0.997, which they 
said might be described, mathematically, as "damn flat". 

Mirrored Copy: http://www.jmargolin.coin/svr/refVref30 kansas.pdf 



Reference 31 - U.S. Patent 4,660,157 Real time video perspective digital map display method issued 
April 21, 1987 to Beckwith, et al. 

USPTO (html): http://patft.uspto.jJov/netaci;i/nph- 

Parser?Sectl^PT01&Sect2^HITOFF&d=PALL&p=l&u^%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=l 
&f^G&H50&sl-4.660.157.PN.&OS^-PN/4.660,157&RS^PN/4.660.157 

PDF: http ://ww w . j margolin.com/svr/refs/ret3 1 4660 1 5 7 .pdf 



Reference 32 - U.S. Patent 5,179,638 Method and apparatus for generating a texture mapped 
perspective view issued January 12, 1993 to Dawson, et al. 

USPTO (html): http://patft.uspto.gov/netacgi/nph- 

Parser?Sectl==Fl'01&Sect2==HI'l'OFh , &d=PALL&p=l&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r^l 
&fr-G&l-50&s 1 =5,1 79.638.PN .&0S=PN/5, 1 79,638&RS-PN/5 J 79.638 

PDF: http ://w w w . i marg olin.com/svr/ rgf s/ ref 32 5 1 7963 8 ,pdf 



Reference 33 - U.S. Patent 4,884,220 Address Generation with Variable Scan Patterns issued 
November 28, 1989 to Dawson et al. 

USPTO (html): http://patft.uspto.gov/netacgi/nph- 

Parser?Sectl-Fl'01 &Scct2-Hri'0h>'&d=PALL&p= 1 &u--%2hnetahtml%2FPTO%2Fsrchnum.htm&r- 1 
&f=G&l=50&sl=4.884.220.PN.&OS=PN/4.884,220&RS=PN/4.884.220 

PDF: http://www.jmargolin.com/svr/refs/ref33 4884220.pdf 



Reference 34 - VCASS: An Approach to Visual Simulation, Kocian, D., 1977, Presented at the 
IMAGE Conference, Phoenix, Ariz., 17-18 May 77. 

Available for purchase from DTIC htt p://www.dtic.mil/srch/doc?collection-t2&id-ADA039999 
Mirrored Copy: http://www.imargolin.com/svr/refs/ref34 vcass.pdf 



A124 



Case 3:09-cv-00421 -LRH-VPC Document 1 -1 Filed 07/31/09 Page 1 24 of 1 28 



57 

Converted to text using OCR (with the paragraphs numbered): 
http://www.jmargolin.com/svr/refs/ref34 vcass.htm 

Reference 35 - The earliest known description of the invention that became U.S. Patent 5,566,073 Pilot 
Aid Using A Synthetic Environment, http://www.imargoiin.com/svi7refs/ref35 pilotdoc.pdf 

Reference 36 - U.S. Patent 5,566,073 Pilot Aid Using A Synthetic Environment issued October 15, 
1996 to Margolin 

USPTO (html): http://patft.uspto.gov/netacgi/nph- 

Parser?Sect1-PT01&Sect2=HlTOFF&d=PALLAp=l&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r^l 
&f=G&l=50&sl=5.566.073.PN.&OS-PN/5.566,073&RS-PN/5,566,073 

PDF: http://www.jmargolin.com/svr/refs/ref36 5566073.pdf 

Reference 37 - U.S. Patent 5,904,724 Method and apparatus for remotely piloting an aircraft issued 
May 18, 1999 to Margolin 

USPTO (html): http://patft.iispto.gov/netacgi/nph- 

Parser?Sectl"PTQl&Sect2^Hri'OFF&d=PALL&pr-l&u-%2Fnetahtml%2FPTO%2Fsrchnum.htm&r^l 
&f=G&l=50&s 1 =5.904.724.PN.&OS=PN/5.904,724&RS=PN/5,904.724 

PDF: http://www.imargolin.com/svr/refs/ref37 5904724.pdf 

Reference 38 - U.S. Patent Application Publication 20080033604 System and Method For Safely 
Flying Unmanned Aerial Vehicles in Civilian Airspace 

USPTO (html): http://appft I .uspto.gov/netacgi/nph- 

Parser?Sectl-PT02&Sect2-HlTOFF&u=%2Fnetahtml%2FPTO%2Fsearch- 

adv.html&r^l8&p^lM^G&l=5Q&d=PG01&Sl=%22svnthetic+vision%22&OS=%22svnthetic+vision% 
22&RS=%22synthetic+vision%22 

PDF: http://www.jmargolin.com/svr/refs/reB8 pg3604.pdf 



Reference 39 - Letter sent to Optima Technology Group by Rapid Imaging Software attorney Benjamin 
Allison, dated October 13, 2006. http://www.imargolin.com/svr/rcfs/ref39 ris.pdf 



Reference 40 - NTSB Incident Report on crash of Predator on April 25, 2006, northwest of Nogales, NM. 
NTSB Identification CHI06MA121 

http://www.ntsb.gov/ntsb/brief.asp7ev id=20060509X0()53 1 &key=%20 1 
Mirrored Copy: http://ww w.j margolin.com/svr/refs/ref40 ntsb.pdf 



.end 
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Jed^Margolin 

From: "Brett Davis" <davis@auvsi.org> 

To: "Jed Margolin" <auvsi@jmargolin.com> 

Sent: Friday, January 30, 2009 4:25 PM 

Subject: RE: Hi from Brett at AUVSI 

Mr. Margolin, 

Thank you for your response to the article that we discussed. 

While I'm not disputing your technical points, for me the point of the story was, as it said early on, to "focus on select 
systems that were important enablers toward UAS synthetic vision systems." If the terminology is rather loosely applied in 
the story that's probably my fault as much as anyone's, but I feel it's sufficient for the purposes of an overview story in the 
magazine. I also don't think that having the story published should interfere with your legal claims as it's not in any way a 
legal document. 

My inclination is not to run anything else in the magazine. Editors generally feel that it's confusing to refer readers off-site 
to responses to articles that ran some months ago. 

That said, I would recommend that you post your links and whatever statements you'd like to make in our Forum section, 
which is open to all, not just members. It's reachable via a link on the homepage. That will give you a little more "room to 
roam" in terms of posting your explanation. In the future we hope to have a web-based magazine display that will allow 
comments, but we're not there yet. 

Thanks again, 
Brett 

Brett Davis 

Editor 

AUVSI 

2700 South Quincy Street Suite 400 
Arlington, VA 22206 
davis@auvsi.org 



Don't Miss... 

AUVSI's Unmanned Systems Program Review 2009 
February 3-5, 2009 

Mandarin Oriental - Washington, DC, USA 

For more information visit: http://www.auvsi.org/programreview/ 



From: Jed Margolin [mailto:auvsi@jmargolin.com] 
Sent: 2009-01-19 19:19 
To: Brett Davis 

Subject: Fw: Hi from Brett at AUVSI 
Mr. Davis. 

Did you receive the email I sent you on 1/8/2009? 
Assuming you did, what did you think about my response? 
Jed Margolin 
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Original Message 

From: Jed Margolin 
To: Brett Davis 

Sent: Thursday, January 08, 2009 1 1 :34 AM 
Subject: Re: Hi from Brett at AUVSI 

Dear Mr. Davis. 

Thank you for permission to post the article Synthetic Vision Technology for Unmanned Systems: Looking Back and 

Looking Forward by Jeff Fox, Michael Abernathy, Mark Draper and Gloria Calhoun which appeared in the December 
2008 issue of AUVSI's Unmanned Systems magazine. 

I expect that the response I have written is too long to print in the magazine (It's 57 pages.) The abridged version is also 
probably too long. (20 pages). I have attached both versions so you can judge for yourself. 

I have done html versions with active links to the references and placed them in a protected directory at: 

http://www.jmargolin.com/svr/auvsLresponseJndex.htm 

Username: Serenity 
Password: Firefly 

{Case sensitive} 

I expect to unprotect the directory soon to make it publicly available. 



Would it be possible for you to print something like the following in the magazine? 

AUVSI member Jed Margolin has taken strong exception to the article Synthetic Vision Technology for 
Unmanned Systems: Looking Back and Looking Forward by Jeff Fox, Michael Abernathy, Mark Draper and 
Gloria Calhoun which appeared in the December 2008 issue of AUVSI's Unmanned Systems magazine. 

Unfortunately, his response is too long to print here. 

In the interests of fairness we are providing the URL to where he has posted his response on his personal web site: 
http://www.jmargolin.com/svr/auvsi response index.htm 

The posting of this URL does not imply AUVSI's endorsement of Margolin's opinions. Mr, Margolin's opinions are 
his own. 

It should be noted that his opinions about the history and the future of synthetic vision are markedly different from 
those of authors Fox, Abernathy, Draper and Calhoun. 

Sincerely yours, 
Jed Margolin 



Original Message 

From: Brett Davis 

To: auvsi@jmargolin.com 

Sent: Tuesday, January 06, 2009 6:42 AM 

Subject: Hi from Brett at AUVSI 

Mr, Margolin, 

We spoke a bit in December about the synthetic vision afljif^gl said ! would get you a PDF copy but I still haven't gotten 
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the final PDF from the printer, although I expect to get that today. Once I have it I can pull that article out and send it 
along. 

It isn't possible for us to post it openly on our website (yet, anyway; that's being redesigned}. You have permission to 
post it on yours if you will include a link back to our site. 

Thanks! Stay tuned, I'll send that along when I get it. 

Brett 



Brett Davis 

Editor 

AUVSI 

703-845-9671 x 208 
571-480-1007 (mobile) 
davis@auvsi.org 



A129 



7/26/09 



